Introduction

In recent years, measuring and improving quality of care has been a thriving area of interest both in North America and internationally. High-quality data which are a result of good documentation practices are essential not only for patient care but for monitoring quality of care and implementing quality improvement strategies. Despite the increasing use of medical records to measure quality of care, studies have shown that the validity of medical records is suboptimal [1,2,3].

Medical records are a key source of information about care processes (e.g. diagnosis, treatment, referral, prescribing) and clinical outcomes (e.g. morbidity, mortality, health-related quality of life) [4, 5]. However, their accuracy may be limited by time pressures and delayed recording which can lead to poor documentation [1, 3, 4, 6]. Other potential drawbacks are the lack of documentation standards [7, 8] and the fact that some of the data recorded in medical records depend on patient self-reported information such as symptoms and lifestyle behaviors [1, 3]. Self-reported data sources such as patient self-administered questionnaires (SAQs) have some advantages over medical records. They are developed in line with research objectives and data generated by SAQs provide insight into the patient’s point of view [9]. However, patient SAQs are dependent on the amount of information the patient remembers, considers relevant, and is willing to share [10, 11].

Several studies have assessed concordance between information collected through SAQs and medical records, but have focused on diagnoses [12], health services utilization [13], medication [14], and specific symptoms [15] or diseases [7]. In a primary care setting, particularly in the context of an intervention program aimed at improving quality of care, little is known on the concordance of care processes evaluated using medical records and patient SAQs. As part of a participatory research program (TRANSIT study) dedicated to TRANSforming InTerprofessional clinical practices to improve cardiovascular disease (CVD) prevention in primary care, members of the primary care community developed a set of 81 process indicators (TRANSIT indicators). Among those, 10 indicators were evaluated through both medical record review and patient SAQs. The objectives of this study are to assess the concordance of care processes evaluated through medical record review and patient SAQs using the TRANSIT quality indicators and identify patients’ sociodemographic and clinical characteristics associated with agreement between the two data sources.

Methods

Study design

The TRANSIT study is a three-phase participatory research program aimed at supporting continuous improvement of CVD prevention in primary care. Details on the study design of each phase are described elsewhere (Phase I [16], Phase II [17], and Phase III [18]). The current study is a secondary analysis of the data collected in the cluster randomized controlled trial (Phase III) and has been approved by the research ethics board of the Centre de santé et de services sociaux de Laval (CSSSL) (2013–2014/04–02). Every participating clinician and patient signed an informed consent form.

Study population

Family medicine groups (FMGs) were recruited in the large urban area of Laval, Quebec, Canada. To be eligible, each FMG had to build an interprofessional facilitation team (IFT) composed of voluntary participating clinicians. The role of the IFT was to promote the implementation of the program among colleagues inside and outside the FMG. It was required that each IFT include at least one nurse, two physicians, and one community pharmacist, as well as one nutritionist, kinesiologist, or psychologist. Some FMGs invited their administrative staff to join the IFT, but this remained optional. Pharmacists, nutritionists, kinesiologists, and psychologists could be recruited from outside the FMG. Each FMG was asked to recruit 100 participating patients. To be eligible patients had to meet the following criteria: 1) have 18 years old and older; 2) be at moderate or high risk of CVD as evaluated by the Framingham CVD risk score [19]; 3) have hypertension, dyslipidemia or diabetes that was either uncontrolled or for which pharmacotherapy had initiated in the past 12 months; and 4) have at least two other chronic diseases, excluding CVD and CVD risk factors. Patients were excluded if they did not meet the 4 inclusion criteria or if they received home care services.

A total of eight FMGs, 98 clinicians and 759 patients participated in Phase III of the TRANSIT study. Six FMGs including 590 patients were randomized into the “facilitation” group while two FMGs including 169 patients were randomized into the “passive diffusion” group. Four FMGs used paper medical records while the other four used paper and electronic medical records (EMRs).

Quality indicators

The TRANSIT indicators were developed through a rigorous participatory process involving researchers, health care managers, clinicians, patients, and family members (see Additional file 1). The full set includes 81 process indicators (see Additional file 2: Table S1) that are evaluated using medical records (n = 60), pharmacy renewal charts (n = 4), and patient SAQs (n = 17). Of the 17 indicators documented through patient SAQs, ten were also documented through medical record review. These indicators read as follows: 1) instructions for home blood pressure monitoring (indicator [IND] 29,30); 2) instructions for home glucose monitoring (IND49,50); 3) education for self-management education for diabetes (IND58,59); 4) referral to a clinician other than a physician and a nurse (IND66,67); 5) referral to a group class offered by the local care center (IND70,71); 6) referral to a community resource (IND73,74); 7) discussion on lifestyle habits during a meeting with the nurse (IND79,80); 8) discussion of the impact of chronic health conditions on the patient’s quality of life (IND82,83); 9) establishment of personal targets for lifestyle changes (IND84,86); and 10) use of the TRANSIT health booklet (IND88,89). The ten pairs of indicators were used in this study to evaluate the concordance of care processes between medical records and patient SAQs. Reliability of the indicators documented through medical record review (IND30, IND50, IND59, IND66, IND70, IND73, IND79, IND82, IND84, and IND89) was previously assessed in a psychometric analysis [20, 21]. Test–retest reliability was almost perfect while inter-rater reliably was substantial to almost perfect [22].

Data collection

Five research assistants attended a two-week training session described in the Additional file 1. Once the training was completed, the research assistants documented the TRANSIT indicators retrospectively over the 14 months prior to the end of the TRANSIT study. For each patient, they assessed whether the care processes were in compliance (yes/no) with the indicators or were not applicable.

A patient SAQ was mailed to patients at baseline and at the end of the study (t14) with a postage-paid envelope for returning the completed version of the questionnaire. Clear instructions on how to complete the patient SAQ were found on the first page of the questionnaire. Patients could communicate with a research assistant by phone if they had any questions concerning the questionnaire. The patient SAQ sent at baseline (n = 101 items) was used to assess quality indicators and collect patient characteristics while the one sent at t14 (n = 97 items) was used to collect data on quality indicators. Items included in the questionnaires were mainly close-ended multiple choice and short open-ended questions.

Statistical analyses

Baseline characteristics of participants were described using means and standard deviations for continuous variables as well as numbers and proportions for categorical variables. At t14, the compliance (yes/no) to each indicator (individual compliance scores) as well as the mean compliance (number of compliant indicators/number of applicable indicators) to all indicators of a category (subscale compliance scores) and to the complete set of indicators (overall compliance scale score) were established.

The concordance of care processes was determined by assessing correlations between results obtained using medical records and those obtained using patient SAQs. Patients with missing data for care processes in either data source during the 14 months preceding t14 were not included in the respective statistical analyses. Percentage of concordant evaluations and prevalence-adjusted bias-adjusted kappa (PABAK) coefficients [23] were computed for each pair of indicators while intraclass correlation coefficients (ICCs) and 95% confidence intervals (CI) were computed for subscale and overall compliance scale scores. PABAK coefficients were used because, unlike kappa coefficients, they are not influenced by prevalence and bias between raters [23]. PABAK values less than 0, between 0.00 and 0.20, between 0.21 and 0.40, between 0.41 and 0.60, between 0.61 and 0.80, and greater than 0.81 indicate poor, slight, fair, moderate, substantial, and almost perfect agreement, respectively [22]. ICCs values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 indicate poor, moderate, good, and excellent agreement, respectively [24].

To better understand if patient characteristics could influence self-report, an explanatory analysis was conducted. This analysis was conducted only among patients with compliant care processes in the medical record and no missing data in the patient SAQ. For each corresponding pair of indicators, generalized linear mixed models (GLMM) (binary logistic regression) with a random (cluster specific) intercept were used to identify patients’ sociodemographic and clinical characteristics associated with agreement of the indicator pair [25]. The sociodemographic characteristics included in this analysis were: age, sex, highest level of education completed, current work, and annual gross family income at baseline. The clinical characteristics included as covariates in the models were: intervention program group, CVD risk category estimated by the Framingham CVD risk score [19], smoking status, body mass index (BMI), diagnosis of hypertension, dyslipidemia, and diabetes as well as the achievement of therapeutic targets for hypertension, dyslipidemia, and diabetes at baseline. An odds ratio (OR) and 95% CI were computed for each pair of indicators. A two-sided alpha level of 0.05 was used to test statistical significance.

Results

Of the 1024 eligible patients invited to participate to the TRANSIT study, 759 (74%) accepted. The patient SAQ response rate was 100% at baseline and 79% at t14. Among the ten indicators evaluated through both medical record review and patient SAQs, seven were applicable to all patients (n = 759 patients), one was applicable to hypertensive patients (n = 596), and two were applicable to diabetic patients (n = 508). Missing data varied from 3% (IND30: n = 15) to 5% (IND66, 70, 73, 79, 82, 84, 89: n = 36) in medical records and from 22% (IND82, 89: n = 166) to 33% (IND29: n = 194) in patient SAQs.

As reported in Table 1, baseline characteristics of the three sets of patients were similar in terms of age (all patients: 62 years; hypertensive patients: 63 years; diabetic patients: 62 years), sex (54%; 52%; 52%), and other sociodemographic and clinical characteristics. Compared to the total population, hypertensive and diabetic patients were less likely to have an annual gross family income higher than $50,000 (36%; 33%; 31%). Diabetic patients were also more likely to be at high risk of CVD (83%; 87%; 96%), to have uncontrolled hypertension (64%; 64%; 70%), and to have a BMI ≥ 30 kg/m2 (57%; 59%; 64%). However, they were less likely to have uncontrolled dyslipidemia (58%; 54%; 47%) than the total population.

Table 1 Baseline characteristics of the TRANSIT study’s participants
Full size table

As reported in Table 2, in the “hypertension management” category, 50% of patients received instructions regarding blood pressure monitoring based on patient SAQs (IND29) compared to 15% based on medical records (IND30). In the “diabetes management” and the “interprofessional collaboration” categories, the percentages of patients reporting compliant care processes were also higher compared to the levels of compliance obtained using medical records. For the category “motivational interviewing and support for healthy lifestyle change”, levels of compliance based on patient SAQ and medical records were similar for the indicators “meeting with the nurse to discuss lifestyle habit” (77 and 78%) and “use of the TRANSIT health booklet” (41 and 46%). However, compared to medical records, compliance assessed by patient SAQ was lower for the indicator “impact of chronic health conditions” (18 and 34%) and higher for the indicator “personal lifestyle changes target” (76 and 55%). Overall, for seven out of ten indicators, the compliance as assessed by patient SAQ was higher than the compliance assessed by medical records. The percentage of concordant evaluations for each pair of indicators varied from 47 to 83%, with PABAK coefficients ranging from − 0.06 to 0.66.

Table 2 Concordance of individual compliance scores to TRANSIT indicators between data sources
Full size table

As reported in Table 3, subscale compliance scores (95% CI) obtained using patient SAQs varied from 41% (38–44) to 78% (74–81) while subscale compliance scores obtained using medical records varied from 15% (12–18) to 53% (50–56). Overall compliance scale scores obtained using patient SAQs and medical records were equal to 51% (49–54) and 37% (35–39), respectively. Concordance between patient SAQs and medical records, as measured by ICCs (95% CI), varied from 0.18 (0.08–0.27) to 0.46 (0.40–0.52) for subscale compliance scores and was equal to 0.46 (0.40–0.53) for the overall compliance scale score.

Table 3 Concordance of subscale and overall compliance scale scores to TRANSIT indicators between data sources
Full size table

As reported in Table 4, to examine the association between patients’ characteristics and the reporting of compliant care processes by the patient among those with compliant care processes based on the medical record, GLMM were used. Indicators related to hypertension management (IND29,30), diabetes management (IND49,50 and IND58,59), and interprofessional collaboration (IND66,67; IND70,71; and IND73,74) were not included in the explanatory analysis due to the large number of missing data and the high prevalence of noncompliant care processes in the medical record. For the four indicators related to motivational interviewing and support for healthy lifestyle change, no sociodemographic characteristics were significantly associated (increased odds: OR > 1 or decreased odds: OR < 1) with agreement between medical records and patient SAQ. In terms of clinical characteristics, smoking (OR = 0.38 [0.18–0.87]) was associated with decreased odds of agreement for the indicator related to a meeting with the nurse to discuss lifestyle habits (IND79,80) while a diagnosis of hypertension (OR = 3.36 [1.12–10.04]) was associated with increased odds of agreement for the indicators related to a personal lifestyle change target (IND84,86).

Table 4 Generalized linear mixed models to identify patients’ characteristics associated with concordance between data sources
Full size table

Discussion

To our knowledge, our study is the first to assess the concordance of care processes evaluated using medical records and patient SAQs in the context of an intervention program aimed at improving quality of care. Quality of care, as measured by the TRANSIT indicators, varied substantially between medical records and patient SAQ. Concordance between the two data sources was poor for the subscale and overall compliance scale scores. Seven out of ten pairs of indicators had lower individual compliance scores when evaluated through medical record review. Three pairs of indicators related to care processes delivered exclusively by the nurse had higher individual compliance scores when assessed using medical records. Except for two clinical characteristics, (i.e. smoking status and diagnosis of hypertension), GLMM showed that, among patients who received care based on the medical record, agreement between the two data sources were not affected by patients’ sociodemographic and clinical characteristics.

Agreement varied substantially between the ten pairs of indicators and was poor for subscales and overall compliance scale scores. These findings are supported by another study evaluating the concordance between medical records and patient self-reports for diagnoses, clinical services delivered, counselling and referrals as well as medication use [14]. As in our study, authors found substantial variations both across and within domains of medical care. However, compared to the overall compliance scale score obtained in our study (ICC [95% CI]: 0.46 [0.40–0.53]), Tisnado and al. concluded that the total concordance between the two data sources was fair to good (kappa = 0.5 [0.5–0.6] and total agreement = 80%). This could be explained by the coefficients used to evaluate concordance (ICC versus kappa) and the domains being assessed in each study.

Compared to the indicators evaluated using patient SAQs, most indicators evaluated through medical record review had lower individual compliance scores, suggesting that clinicians may not consistently record care processes in the medical record. When carefully examining the ten pairs of TRANSIT indicators used in this study, we noticed they were all related to counseling and referrals to some extent. Other studies have reported that these interventions are poorly documented in medical records compared to patient surveys. An observational study including 4454 patients treated by 138 physicians showed that the sensitivity of medical records was low for measuring health habit counseling and moderate for physical examination, laboratory testing, and immunization [2]. Another study conducted in an academic family practice clinic also concluded that the most common physician-patient disagreements concerned counseling or treatment procedure [26]. Finally, a study including 1270 patients sampled from 39 medical organizations found that, at the domain-level, counseling and referrals had the worst concordance [14].

Three indicators in the category “motivational interviewing and support for healthy lifestyle change” had higher individual compliance scores when they were documented using medical records. These three indicators (IND79, IND82, and IND89) evaluated the compliance of care processes that were strictly provided by nurses. These results suggest that nurses may be more rigorous when documenting their interventions in the medical record. This finding is supported by a study aimed at evaluating the documentation of patient pain in the emergency department [27]. Indeed, in this study, nurses were 2.2 times more likely to document pain assessment after therapy than physicians (30% versus 16% p < 0.001) [27]. Another study comparing nurses’ and physicians’ documentation of functional abilities of older patient in acute care showed that nurses took more responsibilities in the documentation of the impairment of both Activities of Daily Living and Instrumental Activities of Daily Living [28].

In our study, sociodemographic characteristics were not associated with agreement between medical records and patient SAQ. Some studies found comparable results [9] while others found differences in the odds of agreement by patient demographics such as age, sex, and level of education [11, 12, 29,30,31,32]. In terms of clinical characteristics, smoking was associated with reduced agreement between medical records and patient SAQs while a diagnosis of hypertension was associated with increased odds of concordance between the two data sources. Patient motivation, which is influenced by a patient’s physical and psychological state, has been found to affect reporting [33]. In our study, hypertensive patients might have been motivated to change their lifestyle habits. This might explain why they were more likely to accurately report a personal target for lifestyle change. Alternatively, sensitive topics such as weight have been found to affect agreement between patients’ surveys and medical records [14, 34]. In our study, smokers might have omitted to report meeting with the nurse to discuss lifestyle habits to avoid exposing their non-adherence to recommendations related to smoking cessation.

Strengths and limitations

This study was conducted as part of an intervention program aimed at improving CVD prevention in primary care. It focuses on care processes that consider priorities for action and challenges such as the implementation of collaborative practices and the provision of appropriate support for lifestyle changes [16, 17]. The reliability of the indicators documented through medical record review were previously assessed in a psychometric analysis [20, 21]. Despite these strengths, our study is subject to potential limitations. Only ten indicators in the complete set of TRANSIT indicators were evaluated through both medical record review and patient SAQs. Missing data in patient SAQs (i.e. up to 33% for IND29) reduced the number of patients included in our analyses. Since neither data source could be considered as a gold standard, it was impossible to determine whether it was the patient (overreporting) or the clinician (underrecording) who was responsible for the discrepancy between the patient SAQ and the medical record. Thus, the way in which indicators are rated (yes/no/not applicable) made it impossible to know if care processes did not comply with the indicators or if the information needed to evaluate the compliance was not found in the medical record during the study period. While this study was conducted in a complex population of multimorbid patients at moderate or high risk of CVD with at least one uncontrolled CVD risk factor, the findings might not be generalizable to the population treated in primary care.

Conclusions

To our knowledge, this is the first study evaluating the concordance of care processes based on medical records and patient SAQs in the context of an intervention program aimed at improving CDV prevention in primary care. Our results suggest that medical record and patient SAQ overall agreement is poor. Therefore, when developing and evaluating the effectiveness of quality improvement strategies, researchers must acknowledge that care processes may not be consistently recorded in medical records. They must also be aware that the evaluation of the quality of care may vary depending on the source of information, the clinician responsible of documenting the interventions, and the domain of care (e.g. treatment, diagnosis, management, medication, counseling, referrals, etc.). Our results also suggest that patients’ sociodemographic and clinical characteristics do not affect the reporting of care processes.