The US healthcare system must address the needs of an aging and increasingly chronically ill population in the setting of unsustainable healthcare spending. Complex patients, often burdened by multiple chronic conditions and psychosocial issues, have more frequent interactions with healthcare systems and are at higher risk for poor health outcomes and avoidable acute care utilization.14 Despite advancements in team-based and non-visit-based approaches to care, the primary care teams that bear disproportionate responsibility for the management of these patients remain overstretched and under-resourced to address their needs. Given the concentration of health expenditures in a subset of complex patients,5 improving the efficiency of care delivery to this group may help contain costs.612

Identifying complex, high-risk primary care patients is an important first step that allows healthcare systems to better allocate resources and target interventions. Current quantitative methods for identifying the complex patients at highest risk for suboptimal future clinical quality and utilization outcomes rely primarily on diagnosis-based and utilization-based algorithms to predict future utilization.1323 These tools miss clinical characteristics that are not present in billing data and may not capture non-clinical contributors to patient complexity.

We previously characterized patient complexity from the physician’s perspective by having primary care physicians (PCPs) review lists of their own patients and identify patients they considered complex. Patients identified as complex by these PCPs had only modest overlap with patients identified as complex using traditional risk-stratification methods.24,25 Clinicians consider medical, behavioral, and socioeconomic complexity domains when identifying complex patients;24 however, asking clinicians to assess each of their patients is impractical, and current population-level approaches generally do not take this comprehensive view. Approximating empiric physician assessment of patient complexity using quantitative approaches might serve as a useful proxy and allow population-level identification of high-risk patients that also require significant effort for primary care teams to manage.

In this analysis, we: 1) developed a predictive model for a physician’s qualitative assessment of their patients’ complexity [estimated physician-defined complexity (ePDC)], and 2) evaluated how patients identified as complex by ePDC compared with those identified by two commonly used risk-stratification approaches in a cohort of primary care patients followed over 4 years. We hypothesized that ePDC would more effectively identify patients with suboptimal outpatient clinical quality and acute care utilization compared to established risk-stratification approaches.


Patients and Setting

We conducted analyses in a cohort of 143,372 adult patients receiving primary care within the Massachusetts General Hospital Primary Care Practice-based Research Network from 2005–2011. We used 3 years of data for predictive model development (2005–2007) and followed patient outcomes over the subsequent 4 years (2008–2011). We obtained all cohort data from an electronic data repository containing demographic, clinical, appointment, and billing data.

Developing and Validating a Predictive Model for Estimated Physician-Defined Complexity

We used an established cohort in which PCPs reviewed 4302 of their own randomly selected adult patients and subjectively identified 1126 patients (26 %) as complex in response to the question, “In your view, do you consider this patient a ‘complex patient’?”24,25 We derived candidate variables (eTable 2) for modeling from demographic, diagnostic, procedure, medication, laboratory, and prior primary care and acute care utilization data. Using data from 2005–2007, we randomly split this cohort into two subsets: two-thirds for model development and one-third for model validation. From the development subset, we generated 1000 bootstrap samples. For each sample, we ran separate logistic regression models with backward elimination (significance at the 0.05 level) to identify predictors for ePDC, stratifying our models by patient age (<45, 45–64, and ≥65 years) to account for differences in the relative contribution of different complexity domains by age category.24,25 We included only those variables chosen in greater than 20 % of bootstrap samples. We evaluated model discrimination using c-statistics and calculated model test characteristics: specificity, sensitivity, and positive and negative predictive value. C-statistics were > 0.76 for all models (0.82 for < 45 and 45–64 years and 0.77 for ≥ 65 years). Test characteristics and information on the final, age-stratified predictive models are available online (eTables 1&2). For our analysis, we selected the model cutoff for ePDC with the highest overall accuracy (e.g., agreement between model prediction and PCP qualitative assessment).

Applying Complexity Measures to the Population

We compared ePDC to outpatient Charlson score26 (OCS) and a proprietary, commercial risk predictor (CRP) used at our institution. We used 3 years of ICD-9 diagnosis codes from outpatient visits, procedures codes, problem lists, laboratory and medication data to define conditions and generate an OCS for each patient. The CRP employs a methodology based on Episode Treatment Groups27,28 and 12 months of preceding billing data, including diagnosis and procedure codes to predict total cost of care in the subsequent year.

We defined complexity thresholds for each algorithm using: 1) a risk score threshold for ePDC that reflected maximal model accuracy in the development data set (14.5 % complex); 2) an OCS threshold of ≥ 4 (11.7 % complex); and 3) a risk score threshold for the CRP matching the proportion of complex patients identified using ePDC (14.1 % complex). Using an OCS threshold of ≥ 3 (18.7 % complex) did not change the direction of any of our findings.

Clinical Quality and Utilization Outcomes

For all eligible patients during the 2008–2011 follow-up period, we determined the start of follow-up time based on test eligibility and censored follow-up time based on the first of any of the following events: death, test ineligibility, or 3 years after the last clinic visit. We calculated the percentage of time (total number of days for which an eligible patient was unscreened divided by the total number of eligible person days in the follow-up period) for which breast, cervical and colon cancer screening was incomplete. We calculated the percentage of time (total number of days for which an eligible patient had an out-of-range test divided by the total number of eligible person days in the follow-up period) for which the Hemoglobin A1c (HbA1c) was ≥ 9 % (for patients with diabetes) or the low-density lipoprotein (LDL) was ≥ 130 mg/dl (for patients with diabetes and cardiovascular disease, based on diagnoses of coronary artery disease, cerebrovascular disease and/or peripheral vascular disease), using the last lab value carried forward. We calculated the number of different utilization events per person year by dividing the number of primary care outpatient visits and “no show” visits, emergency department visits, or hospital admissions by total follow-up time for all patients in the complexity cohort. Finally, we calculated 30-day readmission rates within each complexity cohort.

Statistical Analysis

We used chi-square or t-tests, as appropriate, to compare characteristics of patients and prospective clinical quality and utilization outcomes over 4 years between complexity cohorts. We compared: 1) ePDC-complex groups to non-complex patients, and 2) ePDC complexity groups (ePDC only and ePDC combined with OCS or CRP, respectively), to those that were complex by OCS and the CRP only. Finally, we stratified our analyses for admissions by age (< 45, 45–64, age ≥ 65 years old) to account for potential impacts of age on clinical decision-making.


Patient Cohort

Our baseline cohort of 143,372 adult patients had a mean age of 49 years, was 57.4 % female and 23.1 % non-white race/ethnicity. We identified 19.2 % of these patients as complex by ePDC or OCS; 36.7 % of which were complex by both algorithms (Fig. 1a). One-fifth (20.5 %) of patients were complex by ePDC or CRP; 39.6 % of which were complex by both algorithms (Fig. 1b)

Figure 1.
figure 1

a Complexity as defined by physician-defined complexity (ePDC) and outpatient Charlson Score (OCS).. b Complexity as defined by physician-defined complexity (ePDC) and a commercial risk predictor (CRP).

Comparing Characteristics of Complex Patients by Different Algorithms

ePDC-complex patients were younger and less likely to be commercially insured, English-speaking, male or married, and more likely to live in low socioeconomic status census block groups than OSC-complex or CRP-complex patients. Compared to both OSC-complex and CRP-complex patients, ePDC-complex patients had a greater number primary care visit “no shows” at baseline, were prescribed a greater number of high-risk medications, and had higher rates of chronic obstructive pulmonary disease, diabetes, depression, and psychiatric diagnoses (Table 1).

Table 1. Patient-level Characteristics: Estimated Physician-Defined Complexity vs. Outpatient Charlson Score and Estimated Physician-Defined Complexity vs. the Commercial Risk Predictor

Clinical Quality Outcomes

Over 4 years of follow-up, patients classified as complex by ePDC only spent a greater proportion of time overdue and unscreened for preventable cancers compared to patients complex by OCS and CRP only (Table 2). Differences in the proportion of time unscreened by complexity cohort were as follows (p < 0.001 for all): 1) cervical cancer – 17.8 % (ePDC) vs. 13.3 % (OCS); 19.4 % (ePDC) vs. 11.2 % (CRP); 2) breast cancer – 21.4 % (ePDC) vs. 14.9 % (OCS); 22.7 % (ePDC) vs. 15.0 % (CRP); and 3) colon cancer – 25.9 % (ePDC) vs. 18.7 % (OCS); 27.0 % (ePDC) vs. 18.2 % (CRP).

Table 2. Clinical Quality Outcomes: Estimated Physician-Defined Complexity vs. Outpatient Charlson Score and Estimated Physician-Defined Complexity vs. the Commercial Risk Predictor

Over 4 years of follow-up among patients with diabetes, compared to patients complex by OCS and CRP only, patients complex by ePDC only spent a greater proportion of time with hemoglobin A1c > 9 % (15.6 % vs. 8.1 % for OCS; 15.9 % vs. 6.9 % for CRP, p < 0.001 for both) and LDL > 130 mg/dl (12.4 % vs. 7.9 % for OCS; 11.8 vs. 9.0 for CRP, p < 0.001) (Table 2). For patients with cardiovascular disease, patients complex by ePDC only spent a greater proportion of time with LDL > 130 mg/dL compared to patients complex by OCS and CRP only (10.4 % vs. 5.7 % for OCS; 9.5 % vs 5.3 % for CRP, p < 0.001).

Utilization Outcomes

Over 4 years of follow-up, patients complex by ePDC only made significantly (p < 0.001 for all) more primary care visits (3.78 vs. 2.92 visits per person year (ppy) for OCS only; 3.98 vs. 2.08 ppy for CRP only), had higher rates of primary care “no-shows” (0.45 vs. 0.16 “no-shows” ppy for OCS only; 0.43 vs. 0.15 “no-shows” ppy for CRP only), and emergency department visits (0.21 vs. 0.11 visits ppy for OCS only; 0.17 vs. 0.15 ppy for CRP only) (Table 3).

Table 3. Health Care Utilization Outcomes: Estimated Physician-Defined Complexity vs. Outpatient Charlson Score and Estimated Physician-Defined Complexity vs. the Commercial Risk Predictor

Patients complex by ePDC only had lower overall hospital admission rates compared to patients complex by OCS only (0.15 vs. 0.16 admits/py, p < 0.001). However, when stratified by age, ePDC-complex patients had higher subsequent hospitalization rates in older age groups (0.11 ppy for ePDC only vs. 0.10 ppy for OCS only in the 45–64 year subgroup, p = 0.002; and 0.24 ppy for ePDC only vs. 0.21 ppy for OCS only in the ≥ 65 year subgroup, p < 0.001) (Table 3). Patients complex by ePDC only had lower 30-day readmission rates compared to patients complex by OCS only (13.0 % vs 15.4 %, p = 0.001). Patients complex by ePDC only had lower hospital admission and readmission rates compared to patients complex by CRP only (0.12 vs. 0.16 admits ppy and 12.5 % vs. 14.1 % readmitted, p < 0.001 for both). Compared to patients complex by OCS and CRP only, patients who were complex by ePDC and OCS and ePDC and CRP, respectively, had the highest rates for all utilization outcomes.


We previously showed that primary care physicians (PCPs) take into account patient needs across medical, social, behavioral, and environmental dimensions when identifying their complex patients.24,25 Using data available in health system electronic data repositories, we sought to approximate qualitative physician assessment of complexity by creating a predictive model for PCP-defined patient complexity. We found that this unique measure of estimated physician-defined complexity (ePDC) both successful identified patients that physicians define complex (eTable 1) and prospectively identified patients at increased risk for suboptimal outpatient quality and healthcare utilization outcomes.

This ePDC model had only modest overlap with the outpatient Charlson score (OCS) and a commercial risk predictor (CRP), suggesting that ePDC identified patients with attributes missed by these other methods. Compared to OCS-complex and CRP-complex patients, ePDC-complex patients had a higher prevalence of psychosocial issues, including: mental health diagnoses and proxies for poor adherence and poverty, such as high primary care “no show” rates and residence in neighborhoods with lower socioeconomic indicators.

Over the ensuing 4 years, ePDC-complex patients generally had poorer clinical quality outcomes and higher “no-show” rates compared to OCS-complex and CRP-complex patients. Although the magnitude of the findings in clinical quality outcomes may not be clinically significant within our system due to high screening rates, the effect could be greater in healthcare systems with lower screening rates. Directing health system resources and effective interventions to ePDC-complex patients could improve care for high-risk patients, while enhancing achievement of quality measures used in incentive-based payment approaches and reducing practice income loss from frequent “no-shows.”

ePDC was a stronger predictor of future emergency department utilization compared to OCS-complex and CRP-complex patients, identifying a high-risk group that had a twofold increased utilization rate compared to OCS. Although ePDC performed well in some age strata, ePDC was not generally a stronger predictor of future admissions and readmissions than OCS or CRP. However, patients identified as complex by two different risk-stratification approaches had the highest rates of future acute care utilization. For example, patients complex by both ePDC and OCS had over twofold higher rates of emergency department (ED) visits and admissions and 23 % more 30-day readmissions than patients identified as complex by OCS only. Similar differences were seen for patients complex by both ePDC and CRP. These results suggest that adding ePDC to traditional comorbidity and utilization-based approaches identified a population that was significantly more likely to use acute care services in the near term. This additive effect may be explained by the predictive potential of psychosocial factors captured by the ePDC model. Given the increasing focus on the highest-utilizers of acute care services, ePDC may have significant value in patient identification for intensive care management programs.

Our results have implications for resource allocation and risk-stratification within healthcare delivery systems. Extensive literature demonstrates a link between various diagnosis-based and prior utilization-based measures of complexity, acute care utilization, and costs.1923,2932 Generally, these approaches use administrative claims/billing, and less commonly, electronic medical record data to create risk groupings that predict future utilization and costs.3335 More limited literature demonstrates that adding outpatient utilization information and psychosocial factors improve predictive models for acute care utilization.3537 Prior studies of physician’s ability to predict death or acute care utilization results show mixed results;1318,22 however, our quantitative model, designed to predict physician-defined complexity from increasingly available data, shows promise as an additional tool in identifying and stratifying high-risk patients for population management interventions. Health care delivery systems could replicate our approach in their own context or add proxies for psychosocial complexity that are available in their data repositories to strengthen their risk-stratification approaches. Given that primary care physician review of each patient in their panel is time consuming and difficult to implement broadly, approaches, like ePDC, that combine the PCP perspective, psychosocial risk factors, and traditional medical measures of severity may have broad applicability in primary care risk stratification and resource allocation, and warrant further study.

ePDC may identify patients that are not only at high risk for poor outcomes and utilization of care, but may place a higher burden on primary care teams. Although we were unable to directly assess primary care effort, ePDC-complex patients are more psychosocially complex, have high risk for poor clinical quality outcomes, make more primary care visits, more often “no show” for visits, and present more care coordination complexity due to more frequent acute care utilization. Providing enhanced support to PCPs who manage ePDC-complex patients may reduce clinician burnout, improve clinician experience, and attract new graduates to primary care. ePDC may also be a better marker for PCP panel workload than traditional risk-prediction measures. This suggests the need to further study the use of ePDC-like models in risk adjustment of PCP payments or in the allocation of resources like care management teams or behavioral health supports to clinicians that care for a disproportionate share of ePDC-complex patients.

ePDC is a quantitative predictive model designed to estimate physician-defined complexity within one academic primary care system. Criteria used by physicians to assess patient complexity may be variable. Further validation of ePDC in other settings and populations and research on how physicians identify complex patients is warranted. The accuracy of models will vary depending on available data in other settings. Although our specific ePDC model may not be generalizable to other settings with different physicians and staff support, both the approach used to develop the ePDC model and the addition of psychosocial predictors to traditional risk-prediction models should be. We primarily used electronically available patient data for our analyses, which is limited by missing and unverified data. This may result in some bias; however, this pragmatic approach offers greater generalizability for applications within organized health systems. All clinical quality and acute care utilization outcomes were taken from electronic repositories, and as such, out-of network data were not available potentially biasing results towards the null. Lacking access to claims data, we applied models to our health system’s billing data—a reasonable proxy for the practical use of risk-prediction software in health systems, given the general lack of available claims data. Finally, our approach to identifying patients at risk for poor outcomes was indirect. ePDC and psychosocial predictors should be studied as direct predictors of outcomes in risk prediction models.

This study adds to the literature in several key ways. First, the concept of identifying high-risk patients whom primary care physicians also identify as complex would be a unique starting point for interventions aimed at improving physician experience and efficiency of healthcare delivery simultaneously. Second, using a model such as ePDC to augment existing risk prediction approaches is useful—particularly as healthcare delivery systems seek better ways to identify complex patients for intensive care management interventions. Finally, only limited literature suggests a role for social factors and physician assessment in identifying high-risk patients. While our approach is admittedly indirect in these regards, we do believe it adds to this literature.

In conclusion, a prediction algorithm for physician-defined complexity, which identifies patients that PCPs find complex to manage, was also a predictor for suboptimal clinical quality outcomes and future acute care utilization. Our findings may enhance current risk adjustment and prediction approaches and help identify and risk-stratify complex patients that are both at high risk for suboptimal clinical and utilization outcomes and challenging for physicians to manage. As health systems work to identify complex patients for care redesign in order to improve efficiency of care and reduce costs, our approach informs next steps in the field of risk-prediction.