INTRODUCTION

Initiatives that encourage primary care practices to adopt the patient-centered medical home (PCMH) model of care have proliferated in recent years, under the expectation that offering more coordinated, accessible, patient-centered care will keep patients healthy and out of the hospital—thus saving money while improving patient health. Evaluations of these interventions have attempted to determine whether the PCMH model lives up to this promise.1

So far, the results have been mixed. Some PCMH evaluations have found evidence of improvements in patient and staff experiences, preventive services delivery, patient safety, and reductions in emergency department use.2,3,4,–5 Other studies have identified improved outcomes among patient populations that are sicker than the general patient population (e.g., patients with chronic conditions, children with special needs, older patients).6,7,8,–9 Despite some encouraging trends when looking at health care quality measures, evaluations generally provide mixed or inconclusive evidence of cost reductions.1,4,9,10,11,–12

In an effort to learn more about this delivery reform, the Centers for Medicaid & Medicare Services (CMS) sponsored the Multi-Payer Advanced Primary Care Practice (MAPCP) Demonstration, through which Medicare joined state-run, multi-payer PCMH initiatives underway in eight states. Demonstration payments—ultimately totaling nearly $125 million—supported the provision of patient-centered, comprehensive, coordinated care and enhanced access. More productive provider-patient interactions were expected to lead to improved functional and clinical outcomes, which in turn were expected to produce more efficient health service utilization patterns.13 The MAPCP Demonstration began in late 2011 and ran through 2014—which is the period we analyze in this article—and was then extended until the end of 2016 in five states (Maine, Michigan, New York, Rhode Island, and Vermont).

As part of the demonstration, 849 primary care practices that became certified as PCMHs (using the National Committee on Quality Assurance’s (NCQA’s) standards or comparable state-specific standards) were eligible for supplemental payments from Medicare, Medicaid, and some private payers and also received technical assistance and data reports. Demonstration payments were intended to help practices pay for improvements like hiring care coordinators, expanding their office hours, staffing live after-hours phone lines, or enhancing electronic medical records’ registry capabilities. Outside organizations that supported or supplemented the care delivered by these practices (e.g., community health teams in Vermont, which worked with practices to provide care coordination and other supportive services to patients) also received demonstration payments in five states. Although states designed their own payment models, Medicare payments for practices and other organizations were not supposed to exceed $10 per beneficiary per month, on average; Medicaid and private payers were expected to use a similar approach for determining payment amounts. (An online appendix provides details on MAPCP Demonstration states’ payment models and PCMH practice recognition requirements.)

Given the substantial variation in states’ demonstration designs, experiences, and outcomes, we conducted a qualitative multi-case study to identify the key factors that differentiated states that generated net savings for Medicare from states that did not, since reducing costs is a critical aim of this care delivery model. This article is one of the first of its kind, since in the existing literature, comparisons of different PCMH initiatives have typically focused on identifying the outcomes that the overall PCMH model produces and have not attempted to identify initiative design features associated with successful outcomes.1,2,7,9,14,15

METHODS

This qualitative multi-case analysis uses the “case-ordered predictor-outcome matrix” methodology to relate the key features of states’ MAPCP Demonstration initiatives to a key performance outcome and facilitate the identification of cross-state patterns.16 Here we focus on initiative features related to estimates of net savings to the Medicare program. We identify the initiative features of each MAPCP Demonstration state from comprehensive case studies that summarized findings from a variety of sources, including (but not limited to): researchers’ review of secondary documents (e.g., states’ demonstration applications and progress reports); three annual rounds of semi-structured interviews conducted in late 2012, 2013, and 2014 with state staff, public and private payers, a purposive sample of staff from participating practices, and other stakeholders; and a provider survey fielded in early 2015, which asked about the extent to which demonstration practices engaged in 23 activities associated with the PCMH model.13 (Methodological approaches used to analyze various data sources are described in our final evaluation report; see Chapter 1.13) Data collection was approved by the Institutional Review Boards of RTI International and the Urban Institute, and informed consent was obtained from all interviewees and survey respondents.

To avoid bias in selecting the initiative features we hypothesized might influence a state’s ability to generate net savings, we reviewed the case studies prior to having final estimates of the key outcome. For example, in reviewing the Minnesota case study, it became clear that billing difficulties that prevented providers from receiving demonstration payments was a major issue according to interviewees, so we included billing issues as a feature we should assess in all eight of the MAPCP Demonstration states. Once we had summarized the presence or absence of each initiative feature to be studied, we had the lead researcher evaluating each state review this information to ensure we had accurately characterized each state’s initiative.

We chose net savings for Medicare as our main outcome of interest since it is an important measure to CMS and the public and since it is a metric that all states were hoping to improve through the demonstration. States were classified as having generated net savings if the Medicare beneficiaries attributed to their state’s demonstration practices were estimated to have lower total Medicare spending than beneficiaries attributed to comparison practices, after taking into account the demonstration payments practices received. The comparison practices were not recognized as PCMHs, but were located in comparable geographic areas within or near each of our demonstration states.

We estimated the effects of the demonstration on gross Medicare savings (i.e., savings before MAPCP payments were taken into account) using a state-specific difference-in-differences multivariate regression framework to compare changes in Medicare spending among beneficiaries at demonstration practices to changes among beneficiaries at comparison group practices. (Details of this analysis are available in our final report; see Section 1.2.13) The regression model used to calculate gross savings controlled for a number of beneficiary-, practice-, and area-level characteristics that could independently influence the amount of net savings generated by a state. (These characteristics are described in Section 1.2.5 of our final report.13)

If this model showed that increases in Medicare spending were significantly smaller among beneficiaries at demonstration practices than among comparison group beneficiaries, this meant that the demonstration produced gross savings for Medicare. These savings were weighted by the respective number of demonstration beneficiaries in the sample to produce an aggregate estimate of gross savings (or losses) for each state. Gross savings include total Medicare Parts A and B spending; Medicare Part D spending on prescription drugs was not available to be included in these estimates. (Beneficiaries insured through Medicare Advantage plans were not included in this demonstration and are therefore not included in our estimates.)

Once gross savings were calculated, the amount of MAPCP Demonstration payments were subtracted from these amounts to derive net savings. We include both types of payments made through this demonstration: payments to primary care practices, as well as payments to organizations that supported or supplemented practices’ activities, such as Vermont’s community health teams. Table 1 identifies the net savings generated by each MAPCP Demonstration state; summing the eight states’ financial performance during this period, we find that these states did not generate net savings for the Medicare program overall.

Table 1 Financial Results of the MAPCP Demonstration
Full size table

With the estimates of net savings and the program features hypothesized to be related to this outcome, we constructed a “case-ordered predictor-outcome matrix.” Moving from left to right in this matrix (Table 2), we arrayed states from best to worst performance, with the four states that generated net savings for Medicare clustered at the left and the four states that did not clustered at the right. We use a solid black circle (“●”) to indicate that a characteristic was present in a state, we left table cells blank if a characteristic was absent in a state, and we used a hollow white circle (“○”) when a factor was only partially present. For example, in the “Practices had to be recognized PCMHs when they entered the demonstration” row, the hollow white circle (“○”) in the Maine column indicates that the first cohort of practices that joined the demonstration was given a 6-month grace period to achieve NCQA PCMH recognition, while practices in a second cohort were required to have PCMH recognition upon entry into the demonstration.

Table 2 Factors Present in States that Did or Did Not Generate Net Savings for Medicare
Full size table

One limitation of this study is that not all demonstration states’ net savings or losses were statistically significant, so the ordering in our matrix is based on these best available point estimates of net savings. In addition, state-level factors that we cannot identify may confound the relationships between savings and the features that we can identify. Had these limitations not been present, we would have greater confidence in the importance of the factors identified in our matrix analysis. However, this study is still useful in helping generate hypotheses that could be tested in future studies.

RESULTS

Factors Associated with Generating Net Savings

We considered factors to be associated with generating net savings if they were present in at least three of the four states that generated net savings and absent from at least three of the four states that did not generate savings. (These five factors are shown in the top third of Table 2).

Our first finding was that most states with net savings did not allow late entrants into the demonstration after the start date, while a majority of the states that failed to generate net savings did not have this requirement. In this latter set of states, the number of practices in these states’ demonstrations increased between 50% and 200% over the course of the demonstration. If PCMH practices become more effective over time (e.g., by refining new workflows and care coordinator duties to best fit their practice needs), then it is possible that allowing new, less-experienced PCMHs practices to join throughout the demonstration period may have brought down overall average impacts in states that did not generate net savings.

Once practices had joined a state’s demonstration and were receiving payments, some states gave them a grace period of 6, 12, or 18 months to obtain PCMH certification. We found that most of the states that generated net savings required practices to be certified PCMHs when they entered the demonstration, while states that failed to generate net savings tended to allow practices such a grace period. Practices that were already operational PCMHs at the beginning of their states’ demonstrations may have had a head start on developing and using approaches that could generate cost savings over the course of the demonstration. Meanwhile, practices that were allowed a grace period may have spent the early months of the demonstration trying to understand and adopt the PCMH model at a basic level rather than moving on to more advanced activities like optimizing new care delivery approaches. Consistent with this observation, we found that by the end of the demonstration, practices in states with net savings reported engaging in an above-average share of the 23 PCMH activities included in our provider survey—reinforcing the idea that there may be a link between PCMH maturity and practices’ ability to generate net savings.17

Our third finding is that in most of the states that were successful in generating net savings, all participating payers’ demonstration payment models incentivized performing a consistent set of activities or treating a consistent type of patient (even if payers sometimes paid slightly different amounts), whereas in most of the states that failed to generate net savings, payers used more inconsistent payment models. For example, in Vermont (a state that generated net savings), all payers paid practices higher demonstration payments if they achieved higher NCQA PCMH recognition levels. Meanwhile, in North Carolina (a state that did not generate savings), Medicare paid higher demonstration payments for higher PCMH recognition levels, Medicaid made higher payments for aged, blind, or disabled patients (regardless of what the practice’s PCMH recognition level was), and private payers offered an enhanced fee schedule that was actuarially equivalent to receiving an additional $1.50 per patient per month (regardless of the practice’s PCMH recognition level or the health of a patient).

Another finding related to payment models was that in most of the states that generated savings, participating payers offered practices the opportunity to earn a bonus if they met quality measure performance targets; in contrast, most of the states that failed to generate savings did not offer such bonuses. (Descriptions of how performance bonuses were calculated in these states are included in our online appendix; see Table A-1).

Finally, we found that in most of the states that achieved net savings, demonstration participants tended to receive the payments and bonuses they expected to receive. Meanwhile, in a majority of the states that did not generate net savings, this was not the case. In Minnesota, where the state required practices to submit new claims to fee-for-service Medicare and Medicaid to receive demonstration payments, billing was so cumbersome (and the costs to modify billing systems so steep) that many practices reported it was not worth the effort.Footnote 1 In North Carolina, a change in state vendors resulted in months-long delays in Medicaid demonstration payments. And in Maine, Medicaid changed its payment model for community care teams (a demonstration-funded organization supplementing practices’ activities) midway, switching from making payments for all of a practices’ patients to only funding services for the 5% that were highest risk. In these states, not receiving the payments they expected seemed to make it more difficult for participants to plan, invest in, and implement activities designed to maintain or strengthen their adoption of the PCMH model. This was despite the fact that providers interviewed in all eight states usually reported that demonstration payments were not generous enough to cover the full cost of changes they had made (e.g., hiring new staff).

Factors that May Be Necessary but Not Sufficient to Generate Net Savings

Another set of factors in our matrix was present in at least three of the four states that generated net savings and at least three of the four states that did not generate net savings (see middle third of Table 2). We interpret factors present in a majority of the states that generated net savings as being necessary; however, since these factors were also present in a majority of the states that did not generate net savings, they are not sufficient to distinguish these two groups of states.

These widely observed factors were as follows: a pilot PCMH initiative offered payments to practices before the MAPCP Demonstration; payers participated voluntarily in the demonstration (as opposed to being required to participate by the state); the demonstration enjoyed support from leaders in state government (e.g., state staff were available to work on the demonstration throughout the demonstration period and were not redirected onto other projects); it also enjoyed support from physicians and stakeholders (e.g., these stakeholders viewed their state’s demonstration favorably after helping to develop the PCMH model used in their state); complementary payment and care delivery reforms were underway at the same time as the demonstration (e.g., some payers were entering into shared savings contracts that incentivized practices to more closely manage high-cost patients); practices were required to meet NCQA’s PCMH practice recognition standards (as opposed to some other entity’s standards); and care coordinators tended to focus on high-risk patients.

Factors Not Associated with Net Savings

There were several factors whose presence or absence in the MAPCP Demonstration states did not suggest a relationship with states’ net savings performance (shown in the bottom third of Table 2). Examples of these factors include: the frequency with which practices had to recertify as PCMHs (which ranged from every 1 to 3 years); whether or not states required practices to offer round-the-clock access to care; whether or not states paid other organizations to support practices (e.g., through ongoing data analytics or staff training) or to supplement practices’ activities (e.g., by offering additional care management or counseling to patients); the employment arrangements used for care coordinators (with practices primarily employing them directly vs. third parties like larger health systems embedding their own care coordinators into practices); the professional background of care coordinators (who could be registered nurses or staff with less clinical training, such as social workers or medical assistants).

DISCUSSION

Proponents of the PCMH model may find encouragement in our observation that the longer a practice has been operating as a PCMH, the more likely they are to generate net savings—suggesting that the PCMH model can succeed, but it takes time (and the proper conditions) to do so. Considerable effort and resources have been expended developing PCMH practice recognition standards by a variety of accrediting organizations and payers, designing and funding PCMH demonstrations and pilots, and evaluating these efforts. Our findings suggest that these efforts have not been in vain—but future multi-payer PCMH initiatives should pay special attention to practice entry requirements, and to payment models’ structure, consistency, and disbursal mechanisms, if generating near-term net savings is a priority. Ultimately, our study suggests that it may take more than 3 years for PCMH initiatives to generate net savings—suggesting that longer time horizons may be appropriate when conducting and evaluating PCMH initiatives.

Our finding about the importance of participating payers aligning their payment models to incentivize consistent practice activities could be difficult to implement in practice. Multi-payer initiatives often seek voluntary participation from payers, who may each have their own preferred payment methodology. Payers may also be reluctant to join forces because of anti-trust concerns or to protect proprietary details of their payment model. Initiatives like CMS’s Comprehensive Primary Care Plus model have attempted to get around these obstacles by publicly specifying a recommended payment model and then evaluating payer applications based on the extent to which they align with CMS’s model.18 Some states, in turn, have issued executive orders or passed legislation allowing payers to cooperate as part of multi-payer PCMH initiatives.19 CMS’s State Innovation Models (SIM) Initiative is also attempting to bring payers together at the state level to get them to agree on a consistent payment model to use.20

The issues we identified related to the disbursal of demonstration payments seem quite important. In interviews, demonstration participants complained when they encountered issues that prevented them from receiving agreed-upon payments on time. Payers should recognize that most physicians in the US still work in small practices21 and that disrupting practices’ expected monthly cash flow can have major impacts on their ability to meet payroll obligations, especially when they have hired new staff, such as care coordinators.

Finally, our results suggest that the field still has much more to learn about the optimal design of a PCMH initiative—there is a considerable need for further experimentation to identify additional features that make a difference in a PCMH intervention’s ability to generate favorable outcomes. Future systematic reviews of PCMH evaluations could use similar cross-case analysis methods to identify additional features that are present in successful initiatives and absent in unsuccessful ones.Footnote 2 Future studies could also examine whether larger net savings can be generated when a wider range of providers is incentivized to more closely manage their high-risk patients’ care, such as through accountable care organizations (ACOs) that involve specialists and hospitals.

Limitations

We note that this multi-case study is not exhaustive, and other unobserved or unidentified factors may explain the net savings performance observed in the MAPCP Demonstration (e.g., the share of each practice’s patient panel included in the demonstration, or practice capacity upon entry to the demonstration). Also, several factors included in our analysis are based on findings from interviews conducted with purposive samples of participating practices and other individuals in each state; our interviews are therefore not necessarily representative of the experiences of the full universe of practices and individuals involved in each state’s demonstration initiative. We also note that our study only included eight states; it is possible that a study involving a larger number of participating states may have uncovered different findings.