Background

The introduction of health system reforms in primary care has emphasized a need to evaluate primary care performance and patient experience. In addition to data from chart audits and administrative records, patient and physician surveys provide important information about patient and provider experience. However, recruiting primary care practices, physicians and patients to respond to surveys can be challenging [1, 2], resulting in low survey response rates [1, 3, 4]. The extent to which the survey sample represents the intended population, or survey representativeness, can help to understand the potential for nonresponse bias, that is, a failure to accurately represent the intended larger population [5].

Sampling and recruitment approaches can influence the rate and representativeness of responses [6]. One study compared participants who responded to a survey administered in a waiting room to participants who were emailed the survey, and identified differences in sociodemographic characteristics of respondents [6]. Another study conducted in Ontario emailed a survey to participants in addition to recruiting from the clinic waiting room and found that those who were consecutively sampled from the waiting room were younger and had been patients of the clinic for less time than those who responded to the email survey [7].

Representativeness can be assessed by comparing survey respondents to population data or by comparing late responders to early responders with the assumption that those who respond late are similar to those who do not respond at all [5, 8, 9]. A previous study examined the representativeness of physician and patient participants of an Ontario primary care survey, the Quality and Costs of Primary Care (QUALICOPC). Physician respondents differed slightly from their practice groups, and to a larger extent from other Ontario primary care physicians. Further, patient participants were older, had higher morbidity, and tended to be higher users of the health care system than the general population. One conclusion of this work was that visit-based sampling may have contributed to a biased patient respondent sample [4].

The sampling methods and implementation of surveys may depend on the sampling frame, budget, and setting. Broader or more targeted approaches may be used to recruit physicians, while patients may be recruited from clinic waiting rooms by practice staff, or via email. Two large-scale primary care surveys were conducted in Canada, the QUALICOPC and TRANSFORMATION surveys. We used these to compare the representativeness of physician and patient respondents attained by each approach.

Methods

This study linked physician and patient participant data from the TRANSFORMATION and QUALICOPC surveys to Ontario population based administrative databases held at ICES. ICES is an independent, non-profit research institute whose legal status under Ontario’s health information privacy law allows it to collect and analyze health care and demographic data, without informed consent, housing administrative databases that can be used for health system evaluation and improvement. Both surveys received ethical approval, the QUALICOPC study obtained approval from the University of Toronto Research Ethics Board, and the TRANSFORMATION study received ethical approval from the Ottawa Health Science Network and Bruyère Continuing Care. All participants provided informed consent to participate in the surveys, and for the linkage of survey information and OHIP numbers to ICES. All methods were performed in accordance with relevant guidelines and regulations.

Primary care in Ontario

Ontario is Canada’s largest province by population, with a population of 14.3 million in 2018 [10] and a universal, publicly funded health care system which covers physician primary care services. During the past 15 years, primary care has shifted from primarily fee-for-service (FFS) to 30% patient coverage by physicians in a variety of primary care models [11]. Details about primary care models are outlined in the supplementary table.

QUALICOPC and TRANSFORMATION surveys

The QUALICOPC study was an international survey which aimed to evaluate the performance of primary care systems and meet the need for benchmarking of performance information [12]. In addition to 34 other countries, all 10 Canadian provinces participated in the study which included four surveys: a practice survey, services offered, patient values, and patient experience. In the Ontario arm of the QUALICOPC study, physicians were mailed or emailed invitations through the Ontario College of Family Physicians, to which physicians responded by enrolling on a webpage hosted by the research team. Membership in the Ontario College of Family Physicians is not compulsory for practice, but the college has a membership of over 15,000 physicians and includes most family physicians in Ontario [13]. Practice physicians were asked to collect surveys from ten consecutive consenting patients on a random date (subject to practice considerations) in the subsequent two weeks. Patients were recruited by practice staff. The full complement of surveys was completed in a single day by the majority of practices; exact patient response rates are not available. The Ontario QUALICOPC survey was administered in 2013 and early 2014 [12] and the practice response rate was 3% [4, 12].

The TRANSFORMATION study was a cross-sectional survey which aimed to improve the science of measuring and reporting of primary health care performance. It was conducted in three Canadian regions; the Eastern Ontario Health Unit was the region studied in Ontario and is the focus of this study [14]. A list of practices and physicians created from the College of Family Physicians of Canada and the local health authority was reviewed by a stakeholder advisory group and then used to recruit physician practices. Once practices agreed to participate the research team recruited a consecutive sample of a minimum of 20 patients per practice from the clinic waiting room. Data were collected between 2014 and 2016 [3]. The TRANSFORMATION survey had a practice participation rate of 41.3% and a patient participation rate of 85.5% [3]. See Table 1 for a comparison of the QUALICOPC and TRANSFORMATION surveys.

Table 1 Comparing the QUALICOPC and TRANSFORMATION surveys
Full size table

Measures and data sources

Patient participants from the TRANSFORMATION and QUALICOPC surveys were linked to health administrative databases at ICES using health card numbers and physician participants were linked using unique billing numbers, which were collected as part of the survey and encrypted upon transfer. We used the Client Agency Program Enrolment (CAPE) and Ontario Health Insurance Plan (OHIP) datasets to identify patients of primary care and link them to their corresponding physicians’ identifiers. The Registered Persons Database was used to identify demographic variables, including age, sex, and level of rurality. We measured rurality using the Rurality Index of Ontario which assigns a score between 0 and 100 based on postal code [15], with 0–10 indicating an urban setting, 10 to 40 indicating small urban, and 40 or higher indicating rural. To measure for differences in socioeconomic status we used the material deprivation score from the Ontario Marginalization Index [16]. This score uses census data including the percentage of lone parent families, education level, percentage of those below the low-income measure, and percentage of houses in fair or poor condition, to assign quintiles to each census Dissemination Area (approximately 700 persons) [17], where a quintile value of 5 reflects the greatest magnitude of deprivation [18].

Patient health utilization variables for the year prior to the date of survey administration were sourced from the Discharge Abstract Database for hospitalizations, the National Ambulatory Care Reporting System for emergency department visits, and OHIP billing database for primary care visits. Resource Utilization Bands (RUB) are part of the Johns Hopkins ACG system [19]; they are a ranking system of overall morbidity from 0 to 5 based on epidemiological patterns. We used ICES derived datasets to identify participants with the following chronic disease diagnoses, based on validated case definitions: asthma [20], congestive heart failure [21], chronic obstructive pulmonary disease [22], hypertension [23], and diabetes [24]. .

Physician variables including years in practice, age, sex, and whether they graduated from a Canadian medical school, were derived from ICES Physician Database. CAPE and OHIP databases were used to identify the size of the physicians’ roster, and the primary care model type. These datasets were linked using unique encoded identifiers.

Comparison groups

To compare physician respondents of the TRANSFORMATION survey, we created a group of physicians who worked in the same practices as the TRANSFORMATION survey physicians but did not participate in the study. The survey focused on a public health unit in Eastern Ontario, thus, the second physician comparison group was family physicians working within this geographical area who did not participate in the original study; this third group included those who were in the same practice as the survey respondents. QUALICOPC physicians were similarly compared to others from their practice groups and other primary care physicians in Ontario [4]. To compare QUALICOPC and TRANSFORMATION patient participants to other patients who use primary care we created comparison groups for each survey. We created a group of other patients who saw the responding physician on the day of survey administration. We also created a group of patients who saw each responding physician on four randomly selected dates throughout the year (one date selected in each quarter). The dates were randomly selected using analytic software, excluding the use of weekends, holidays, and days where the physician saw less than 10 patients. The use of randomly selected days was used to account for potential effects of seasonality in terms of volume and type of patients visiting the practice [25].

Statistical analyses

We used standardized differences to examine differences between physician and patient groups because they quantify the magnitude of difference without relying on sample size [26]. Standardized differences are the difference in means of a variable between two groups divided by its standard deviation, and are also used to compare the prevalence of dichotomous variables between two groups [27]. For the purposes of this study, a standardized difference of 0.1 indicated a small but meaningful difference between groups [26]. All analyses were conducted in SAS software, Version 9.4 (SAS Institute, Inc.).

Results

Physician representativeness

In total 47 physicians who completed the TRANSFORMATION survey were linked to health administrative databases and were compared to 69 physicians in the same practice groups, and 219 physicians practicing in the Eastern Ontario Public Health Unit. Participating physicians were similar in age. They included fewer female physicians when compared to the same practice group, but this was comparable to other physicians in eastern Ontario. TRANSFORMATION providers had roster sizes that were more similar to those of their colleagues in the same practice group and tended to be larger than those of other physicians working in the same geographical area. Physician survey participants also included more physicians who worked in community health centre (CHC) practices. They included fewer physicians working in fee-for-service models compared to the geographical area, with higher representation of physicians working in FHTs in the region (Table 2a).

Table 2a Comparing physician participants of the TRANSFORMATION survey to other physicians in the same practice group and physicians in Eastern Ontario Public Health Unit (PHU)
Full size table

QUALICOPC physicians were compared to other physicians in their practices and in the province of Ontario in a previous study that was authored by the authors the current study [4] (Table 2b); this table was used with permission. Briefly, we found that physician respondents were, on average, younger, had fewer years of experience, and consisted of a higher proportion of physicians who attended medical school abroad [4]. Survey respondents also consisted of fewer solo physicians [4].

Table 2b Comparing physician participants of the QUALICOPC study to other physicians in the practice group, and Ontario primary care physicians
Full size table

Patient representativeness

534 patient participants of the TRANSFORMATION survey were compared to 1,523 patients who visited the same physicians on the day of survey administration, and 2,900 who visited the responding physicians on four randomly selected dates throughout the year. Compared to patients seeing physicians on the same day, survey participants consisted of a greater proportion of individuals between 45 and 64 years of age and fewer individuals who were 65 years of age or older. They consisted of fewer individuals with very high morbidity than others seeing the same physician on the same day and throughout the year. They were from more rural areas when compared to the same day, and to a lesser extent, throughout the year. Lastly, survey participants had fewer acute care visits and were less likely to have COPD or CHF compared to both same day and randomly selected day patients (Table 3a).

Table 3a Comparing patient participants of the TRANSFORMATION survey to other patients visiting the practice on the same day and randomly selected dates
Full size table

A total of 1,225 patient participants of the QUALICOPC survey were linked to health administrative databases and compared to 29,885 patients who visited participating physicians on the same day, and 6,828 who visited the same physicians on four randomly selected dates throughout the year. Comparing the QUALICOPC group to those who visited on the same day, all standardized differences were under 0.1. QUALICOPC participants included a lower proportion of people with low morbidity compared to patients from randomly selected days throughout the year. In addition, there was a higher proportion of respondents with asthma in the QUALICOPC patient sample than in those attending on random days (Table 3b).

Table 3b Comparing patient participants of the QUALICOPC survey to other patients visiting the practice on the same day and randomly selected dates
Full size table

Discussion

This representativeness study compared patient respondents to other users of primary care across two primary care surveys that used different sampling and recruitment techniques. We found that patient respondents of the QUALICOPC survey were representative of other patients who accessed primary care on both the day of survey administration and throughout the year. However, patients responding to the TRANSFORMATION study comprised of more people between 45 and 64 years of age, were from rural areas, were lower users of the health care system and had lower incidence of chronic disease. We also found that physicians who responded to TRANSFORMATION differed compared to other physicians in the geographic area in terms of proportion of physicians trained in Canada and included more male physicians than physicians in their practices. QUALICOPC physicians were negligibly different from their practice groups but were found to be younger, included more females, and cared for larger rosters of patients than other physicians in the province [4].

The extent to which physicians differed from the larger intended population is similar to other literature emphasizing the different characteristics between physicians who volunteer for research studies, and those who do not [1, 3]. These studies suggest that physicians who participate are more likely to be female, as in our QUALICOPC physician findings, and work in team-based models of care [4, 28], which was noted in results of both TRANSFORMATION and QUALICOPC physician participants. Other studies have also suggested that it is difficult to recruit physicians working in fee for service models of primary care [28], resulting in the underrepresentation of this group in primary care surveys. This pattern of practices recruited was noted in physician respondents of the TRANSFORMATION survey, but not in the QUALICOPC respondent physicians, where the proportion of physicians working in fee for service models was similar to the proportion within the entire province. The TRANSFORMATION survey invested additional resources into physician recruitment and ensured a response rate of 41.3%, while the QUALICOPC survey sampled broadly but did not invest additional resources in physician recruitment and had a physician response rate of 3%.

The differences between the QUALICOPC and TRANSFORMATION patient representativeness was surprising as both studies used consecutive visit-based sampling, with the former using practice staff to distribute surveys while the latter used survey administrators. The visit-based method of recruitment has been reported to under-represent low users of care and not be representative of housebound patients [29]. In an Ontario study, a waiting room sample of patients was found to be older and more likely to be female than the overall practice population [28]. A similar finding was noted for QUALICOPC patient participants, where they were representative of patients visiting the practice, while TRANSFORMATION respondents were younger compared to other patients visiting their physicians. Similar to TRANSFORMATION, many surveys that recruit from clinic waiting rooms use a research assistant to either conduct all the recruitment, or to explain the study to participants and request informed consent after front desk staff have referred participants to them [7, 30]. In this study, survey distribution by practice staff (QUALICOPC) appeared to generate a more representative respondent sample than distribution by research staff (TRANSFORMATION). Further exploration of the difference between practice and research staff in this context is warranted, including the number of surveys distributed. In addition, the impact of survey recruitment on practice staff should be considered.

The QUALICOPC survey respondents included a higher proportion of individuals from the lowest quintile of material deprivation. The relationship between area deprivation and survey responses has been studied with a view to increasing response rates from these groups, with emphasis on using in person methods to facilitate responses [31]. Information about reasons for declining participation were not collected, thus, further investigation into the intersection of material deprivation and survey response rate is warranted.

Limitations

While we used administrative datasets to derive physician and patient comparison groups, certain limitations must be noted. The variables used included health care utilization, health status, and sociodemographic characteristics. However, we did not include a measure of access to primary care, despite the fact that this is a key performance indicator for this sector, was measured as part of the QUALICOPC survey [3], and would have been relevant given the visit-based approach to sampling. In addition, we did not examine survey outcomes in relation to the characteristics of respondents [32]. Such an approach may have given a more accurate understanding of the effect of representativeness on results. We conducted this study in the province of Ontario, which has a single payer health system that includes physician primary care services; the findings may not be generalizable to jurisdictions where a different structure of primary care exists. Both surveys examined used in person, visit based sampling. Thus, our conclusions are not transferable to mail or email surveys, which have both been associated with different respondent populations in primary care [6, 33]. This limitation is particularly relevant given that the survey data analyzed are now several years old. The use of telehealth has increased over the past few years, and with it, the use of electronic surveys [34, 35]. Understanding representativeness of electronic surveys may help inform methods, and support interpretation of findings in context.

Conclusion

This study contributes to a growing body of research on nonresponse bias in primary care surveys, with specific consideration to practice based sampling approaches. It is possible to recruit a convenience sample of patients in the waiting room who are representative of typical users of primary care. However, representativeness may be affected by the individual recruiting and distributing the surveys. With respect to physicians, targeted and selected recruitment resulted in a higher response rate for the TRANSFORMATION survey, but did not achieve a more representative sample of physicians than the less intensive and less expensive approach did. Our findings emphasize the need to measure and account for survey representativeness through techniques that can be applied during sampling, such as delayed sampling or oversampling [5, 9] of groups that are known to be poorly represented.