Introduction

Despite decades of discussion and experimentation concerning publicly reported healthcare outcomes, most patients and their families cannot ascertain the relative performance of the healthcare professionals into whose hands they put their children’s lives. Such asymmetry of knowledge commonly occurs in business transactions such as buying a used car, the seller always having the advantage, because he/she knows more about a specific car than the buyer can possibly learn prior to the sale. However, in an information-saturated world, consumers can learn much more about car dealerships, restaurants, or hotels’ goods and services than they can about hospitals and healthcare providers. Consumers can rarely obtain comparative, clinically important outcomes information about pediatric health care. Such an asymmetry of information does not serve patients and their families nor does it bode well for pediatric healthcare as an industry or as a mission-driven enterprise. Either healthcare professionals will provide comprehensible healthcare information or someone else will. The current knowledge asymmetry appears as an anomaly in our increasingly interconnected world.

Numerous non-medical entities have already acted to fill the void. U.S. News and World Report (USNWR) publishes annually what is probably the most well-known source of comparative pediatric hospital outcomes information [1]. While some hospitals might complain about the emphasis on “reputation score,” USNWR has incorporated many measures that reflect clinical excellence and clinical efforts to produce high-quality outcomes. USNWR transparently publishes its scoring criteria each year, and one can go to their Website and see how each hospital scored on specialty-specific criteria. The Center for Medicare and Medicaid Services makes available selected outcome measures on its Website, “Hospital Compare,” although many of the measures apply mostly to adult, not pediatric, healthcare facilities [2]. States, such as Minnesota and others, also provide comparative data on healthcare [3], as do commercial sites such as Angie’s List, Consumers’ Reports, Healthgrades.com, and many others. Commercial sites may represent the voice of the consumer but often lack the clinical or analytical expertise to provide families with reliable information on clinical outcomes. This leads to the question of interest, “What information should pediatric hospitals report publicly and how should they report it?” This review seeks to discuss the benefits and risk of outcomes reporting and what several pediatric healthcare organizations are doing to self-report their outcomes, primarily on the Internet.

History of outcomes reporting

Healthcare has a long history of measuring and utilizing clinical outcome information. Semmelweiss [4], Florence Nightingale [5], and Ernest Codman [6] all used comparative clinical outcomes to measure and improve their practice of medicine and surgery over a century ago. And all met with considerable criticism, if not hostility, from their peers and superiors. Semmelweiss lost his job, Nightingale was mocked, and Codman was forced to leave his position at the Massachusetts General Hospital and start his own hospital. Little wonder that these pioneering efforts did not lead to wider outcomes reporting for decades. Hospitals and healthcare and provider organizations have for many years unwittingly or purposefully kept their clinical outcomes private, making it difficult for patients to know what they are purchasing and whether or not they are likely to receive maximum value for their purchase. Although most of these organizations expend great energy and resources to measure, assess, and publish their business outcomes such as revenue and income, they have generally spent much less energy on assessing the quality of the services generating their revenue.

It was not until the late twentieth century that outcomes measurements became publicly and widely available. In 1986, the Healthcare Financing Administration (now known as the Centers for Medicare and Medicaid Services or CMS) published hospital mortality rates. This effort generated an outcry based on methodological flaws, such as inadequate risk adjustment, and met an early termination in 1993. However, other groups began to study common cardiovascular procedural and other clinical outcomes. One of the most successful was the Northern New England Cardiovascular Disease Study Group (NNECDSG), which tracked coronary artery bypass procedure outcomes. The NNECDSG addressed issues of data accuracy, audited participating hospitals, utilized clinical data, and developed risk-adjustment models. Most importantly, their reporting was accompanied by collaborative quality improvement efforts. Other groups such as the Vermont Oxford Network and Veteran’s Administration and states, such as New York, Pennsylvania, Minnesota, and others, have also developed registries of clinical outcomes and to one degree or another made at least their aggregated data public. Professional societies, such as the Society of Thoracic Surgeons [7], also allow the public to view outcomes of commonly performed procedures by hospital. The process of tracking, analyzing, and reporting outcomes either publicly or confidentially seems well underway.

The Northern New England Cardiovascular Disease Study Group

The Northern New England Cardiovascular Disease Study Group (NNECVDSG) serves as a good example of public reporting [8]. They pioneered public reporting of cardiovascular outcomes and detailed their reports to the level of the surgeon. Their efforts began in 1987 as a collaboration of cardiovascular centers in Maine, New Hampshire, and Vermont [9]. They started with coronary artery bypass grafting (CABG) outcomes and have since expanded to include other cardiovascular procedures. They found considerable variation in CABG outcomes despite risk adjustment. However, they did not limit their work to publishing outcomes; they coupled the reporting with collaborative efforts to identify and implement best practices. The NNECVDSG reduced mortality by an impressive 24 % [10].

Why healthcare organizations resist publicizing their outcomes

Getting outcomes information into the public domain has been an understandably difficult challenge and due to cogent concerns. A number of rationalizations have been made to explain why transparency cannot be achieved. These include the following:

Legal concerns

While much has been published about legal issues regarding reporting of specific medical errors or poor outcomes to individual patients, there appears to be no study of the generalized legal risks to hospitals for reporting overall quality (or lack thereof). Of course, all released data must meet privacy guidelines (no identifiable information, etc.). If that criterion is met, and given that revealing errors to the individuals affected has not increased malpractice claims [11, 12], it appears that hospitals or practices should not be concerned about legal ramifications of transparency. Furthermore, on the face of the matter, it seems unlikely that a member of the public at large could demonstrate legal standing to bring a case against a hospital or practice based on composite data.

Validity of data/risk adjustment

Organizations, especially those who have concerns about how their quality data looks, may be reluctant to report and instead take the attitude that “our patients are different” and that without appropriate risk adjustment, they will be perceived as providing lower quality care when that is not in fact the case. Unfortunately, it is also a statistical truism that half of pediatric healthcare organizations must produce below-average outcomes. Each organization will undoubtedly have a strong urge to find a way to report data that never represents themselves as being in the bottom half. Risk adjustment attempts to validate hospital comparisons by adjusting outcomes for the risk factors/differences in the patient population at the beginning of treatment. This remains an imperfect effort. For example, adjusting mortality estimates for admittance through the emergency department might seem reasonable because these patients might be sicker. However, being admitted through the emergency department may have different implications at different hospitals [13]. Furthermore, there may be intrinsic differences in patient populations that cannot be captured by currently known risk factors. Additionally, clinicians have failed to agree on the clinical outcomes of greatest value to patients and families. Therefore, each hospital chooses which outcomes it will report, many of which may not be reported by other hospitals. Goff et al. [14] recently reported on the attitudes of QI leaders on publicly reported CMS measures. The vast majority of comments addressed perceived invalidity, unimportance, or unfairness of the reported measures. These results mirror multiple other publications in other contexts.

Inability of public to comprehend data

Hospitals may have concerns about the public’s ability to understand more generalized data regarding quality of care provided. A white paper from the Joint Commission [15] examined whether consumers could correctly interpret data that compares hospital performance. Test subjects reported that they wanted as much information as possible, but actually providing an abundance of data led to more difficulty with comprehension. The authors tested 24 prototypes of data presentation and found that consumers could generally interpret data, but they were unable to find one method of data display which was universally superior to others. One consistently difficult display was a run chart with multiple hospitals, and one consistent improvement to understanding was to display any graphs with “up” being “good.”

Questions about how the public will use the data (or not)

As discussed in the next section, one may question whether the public will do anything with publicly available quality data. Faber et al. [16] performed a review of the literature and conclude that the ability of public reporting to influence consumer behavior will be limited until a few barriers are overcome which include making the public aware that such data is available, helping the public understand presented data, changing attitudes about leaving a physician or health system with poorer quality of care, and removing financial/geographic/logistical barriers to seeking the highest quality providers.

Benefits of transparency to individual hospitals

A few questions face the institution which is considering increasing its transparency. First, the institution must make a strategic decision as to whether they will be more transparent about costs, quality, or both (e.g., value). Second, the institution must decide whether to be transparent to specific organizations alone (e.g., payers and regulatory agencies) or whether to become openly transparent to everyone including prospective patients, the community, and competitors. Finally, the answers to these questions may depend on the hospital’s aims, and their assessment of how transparency drives those aims. Potentially, the hospital may suspect that transparency to all may affect market share, may motivate internal changes to increase quality and/or costs, or may affect goodwill (the amount of value a company’s reputation adds to overall value).

Improvement in quality of care

Numerous studies have shown an association between public reporting and improved clinical outcomes [17–19]. However, not all studies have been conclusive and a recent Cochrane Review gave only a tepid endorsement of public reporting as a method to improve outcomes [20]. Despite some reservations, public reporting has repeatedly been associated with improved outcomes, especially in cardiovascular procedures. However, transparency can also distort quality improvement efforts, with hospitals working more to improve the measure than the quality of care [21]. Upcoding to inflate the expected mortality, refusal to treat high-risk patients, and increased racial disparities have all been occasionally associated with public outcomes reporting [22–24]. Confidential reporting of outcomes to physicians and organizations may be as effective as public reporting [17]. Public or confidential outcomes reporting will only improve care if it motivates clinicians and healthcare organizations to improve care processes. A recent survey of hospital staff suggested that public outcomes reporting provided a needed impetus for organizations to improve or maintain quality [25]. Indeed, the key function of public outcomes reporting may be to stimulate healthcare organizations to improve the quality of their care, which is reflected in improved outcomes over time. It is difficult to improve what one cannot measure, and competitive measurements may be particularly valuable in improving performance.

One other study of whether transparency might lead to improvement in the quality of care provided was published by Lindenauer [26]. Their study examined facilities which publicly reported quality of care metrics, either with or without pay-for-performance incentives funded by CMS. They found that adding a pay-for-performance incentive led to greater improvement in the quality of care as compared to public reporting alone; however, hospitals without pay-for-performance did show improvement over time, and they were not able to compare the group with public reporting to hospitals with no public reporting. Finally, Hibbard et al. [27] examined the effect of public reporting of hospital quality of care in a specific geographic area. They found that the odds of improvement in quality of care was higher in hospitals with public reporting than hospitals which received non-public report cards, which was greater than hospitals which received no report card (these last hospitals showed no improvement at all). While there were no market shifts with public reporting, the public was able to identify top- and bottom-performing hospitals because of the wide reporting of the quality metrics in the media.

Market share

If one suspects that transparency of quality metrics leads to improved market share, the presumptive mechanism is through competition against hospitals with lower (perceived or reported) quality. However, public outcomes reporting appears to have far less effect on market share than it does on efforts to improve clinical outcomes. Healthcare organizations might worry that if their publicly reported outcomes are inferior to others in their service area, they would lose market share. To date, this concern lacks a strong evidence base.

An excellent study on the question of competition and quality choice, which touches on both the questions of market share and hospital performance, has recently been published by Lewis and Pflum [28]. These authors note that evidence of quality report cards driving consumer choice is limited (also reported by Dranove and Sfekas [29]). They find that competition (which can only take place in the context of transparency to those making a choice) for HMO or MCO patients leads to higher quality of care for all patients in the hospital, while competition for Medicare patients leads to lower quality for all patients. They suggest that this difference arises because of the relative importance placed on quality versus cost by these two segments of the market.

In California and New York, after public reporting, high-mortality, high-complication hospitals experienced only modest transient or no changes in market share or volume [30]. One study reported that one, well-publicized, unexpected death had far more effect on market share than publicly reported hospital mortality data [31]. Furthermore, hospitals’ performance in a specific region may differ little, so that outcomes data appear so similar as to be unhelpful in distinguishing hospitals [32]. These examples notwithstanding, the public’s appetite for comparative hospital performance data may be changing. The current generation of new parents have grown up in the information age of smartphones and wide availability of comparative rankings of everything from vacation spots to vacuum cleaners [33]. It would seem reasonable that although outcomes data have not influenced market share in the past, it may well do so in the future.

Three children’s hospitals’ approaches to transparency

Children’s Hospitals and Clinics of Minnesota

Children’s started publishing clinical outcomes in paper form in 1994. These included only NICU outcomes and were based on Vermont Oxford Network data specific to Children’s. Outcomes from other clinical areas were added in 1998. The published outcomes were given to families at prenatal consultations, on admission, and at anytime families had questions about possible outcomes. Children’s continued various versions of the paper reports until 2010 when hospital-wide outcomes were placed on our Intra-net and Internet sites. The presentation of outcomes was initially organized around the Institute of Medicine’s six domains of quality: safe, effective, patient-centered, timely, efficient, and equitable. However, families had a hard time comprehending the graphs and following the flow of information so Children’s condensed the topics into “Population Health,” “Quality of Care,” and “Service Excellence.” At first, they displayed bar graphs and then later went to run charts in an effort to provide more information. However, families reported that run charts were too complicated. They felt that the outcomes looked like they had been prepared for healthcare professionals and not the intended audience, families. The Family Advisory Council asked us to produce simple tables with comparative information and emphasized that what the hospital was doing to improve the outcomes was much more important to them than the sterile facts. A small sample of the current iteration, the third version of Internet-reported outcomes, appears in Fig. 1a, b. Children’s is currently revising the entire Website, so the presentation of quality data will be taken back to the Family Advisory Council for their revisions and reformatted on the updated Website over the next 6 months. The Website resides at http://www.childrensmn.org/about-us/quality-of-care-at-childrens.

Fig. 1
figure 1

a An example of data displayed on the Children’s Hospitals and Clinics of Minnesota Website. This is one of many metrics publicly released, and clicking on the metric opens a run chart as shown in b. b A run chart of a specific measure from the Children’s Hospitals and Clinics of Minnesota Website.

Full size image

Cincinnati Children’s Hospital Medical Center

The vision of Cincinnati Children’s Hospital Medical Center (CCHMC) is to be the leader in improving child health, and our mission is to achieve the demonstrably best medical and quality of life outcomes, patient and family experience, and value. As a key driver of healthcare transformation to achieve our vision and mission, Cincinnati Children’s developed a transparency strategy in 2006 based on three main premises: (1) Transparency strengthens a culture of inquiry and learning by providing all levels of the organization access to the same performance data and empowering employees to act upon it and improve; (2) sharing information externally promotes an open and collaborative healthcare environment and invites others beyond our geographic area to partner and explore new opportunities to improve child health; and (3) making data available meets and exceeds the expectations of families and helps support their healthcare decisions for their children.

Our approach to incorporate transparency into work of all strategic initiatives led us to a standardized approach to all of our publically reported data. First, we assure that all of the outcomes we report are relevant to families and healthcare providers; the data is checked for accuracy and is accompanied by operational definitions so that other healthcare organizations can replicate the measures if desired. The posted outcomes are accompanied by important patient, family, and/or workforce stories to engage the audience. Finally, each transparently reported measure tells the reader of ongoing process improvement work. Most measures are updated monthly or quarterly based on data availability.

Our externally posted quality measures fall into one of three categories. First, we post our CCHMC system-level measures, a parsimonious set of measures to guide and track our whole system transformation that are aligned with our vision and mission, and reflect our constancy of purpose. System-level measures fall into the categories of patient and employee safety, outcomes and clinical excellence, patient and family experience, access flow and productivity, and team well-being. Second, we post condition-specific measures with comparative data when available. Finally, we report links to our other publically reported data such as Leapfrog, US News and World Report, Ohio Hospital Compare, Joint Commission asthma care ratings, and United Network for Organ Sharing. All measures may be found at http://www.cincinnatichildrens.org/about/quality-measures/default. Examples of the system-level measures are shown in Fig. 2a, b.

Fig. 2
figure 2

a A list of system-level measures transparently reported on the Cincinnati Children’s Hospital Medical Center Website. b Example of a system-level measure in the Access, Flow and Productivity category of system-level measures from the Cincinnati Children’s Hospital Medical Center Website.

Full size image

As a result of our dedication to transparency, Cincinnati Children’s was the recipient of the 2012 Hospital Website Transparency award given by URAC and the Leapfrog Group. CCHMC was one of only two hospitals to receive distinction with honors in this area.

Nationwide Children’s Hospital

Nationwide Children’s Hospital (NCH) had an Internet site with quality data but made a series of decisions from 2008 to 2011 to gradually release this information on the Internet. Initial internal discussions raised issues mentioned above, primarily regarding “managing the brand” and whether revealing our flaws would create public backlash. At this same time, we had an organizational culture emphasis towards improving quality and safety of care, and eventually, it was felt that only full transparency and accountability to the public was consistent with our mission and values. The first metrics were released in the spring of 2009, and in September of 2011, we released our serious safety event rate publicly. The Website (http://www.nationwidechildrens.org/quality-safety) is organized according to our re-interpretation of the IOM’s quality domains which are Keep Me Well, Heal Me, Do Not Harm Me, Treat Me with Respect, and Navigate My Care [34]. Most pages describe a specific measure, with descriptive videos, and include control charts of performance. Data is updated quarterly or more frequently. We have determined that 75 % of our traffic comes from browsers in which our address was typed or bookmarked—presumably people who visit the site regularly or are familiar to our institution, with half of visiting computers being from Ohio. On average, 80 % of the time one of our quality and safety pages is being viewed, the next page viewed is also on our site, indicating that viewers move around the site instead of viewing one page then leaving our domain. Our safety-related pages are visited much more often than non-safety quality-related pages. In summary, we hope that our site is effective for both consumers as well as other organizations seeking information about our performance. Examples of data on our Website are shown in Fig. 3a, b.

Fig. 3
figure 3

a A control chart displaying the adverse drug event rate from Nationwide Children’s Hospital Website. b A control chart displaying the inflammatory bowel disease remission rate from Nationwide Children’s Hospital Website.

Full size image

Conclusion: where do we go from here?

Despite considerable progress in making clinical outcomes available for pediatric healthcare, significant barriers need to be addressed. First, what outcomes are most important to patients and their families? Rarely do hospitals report measures of patient-centered care, rather they report clinical outcomes. The voice of the patient needs to be more prominent in the selection of outcomes measures. Currently, clinical measures lack standardization. Clinicians and administrators usually determine which outcomes will be reported to the public. The variety of outcomes reported and the almost universal favorable standing of the reporting hospital in those reported outcomes suggest that hospitals select outcomes to be reported, in part, based on whether or not they show the hospital in a favorable light. Furthermore, there are over a hundred potential measures endorsed by the National Quality Forum [35]. Which of those best represent clinical quality of child health? Finally, how would a family know which measures should most concern them? Standardization, preferably by a group with clinical expertise and political clout, seems the next most important step to helping our families know the value of the care that they provide for their children. We hope that clinical leaders will take the lead on establishing standard outcomes for public reporting.