Transparency in Pediatric Outcomes Reporting—Reducing Knowledge Asymmetries in Pediatric Healthcare
- 526 Downloads
Hospitals and pediatric practices may be under increasing pressure to be publicly transparent about their quality and safety outcomes. In this review, we examine the literature which addresses the risks and benefits of transparency and provide examples of how our three children’s hospitals have approached becoming transparent to the public about our outcomes. We briefly reviewed the history of healthcare outcomes reporting from Semmelweiss to the present. We also examined how three pediatric hospitals have taken somewhat differing approaches to making their clinical outcomes available on the Internet. Public reporting of clinical outcomes has largely been associated with an improvement in those outcomes. Legal risks and the potential loss of market share based on less than stellar performance have been and are concerns that inhibit some institutions from sharing their outcomes. Based on published experience, those risks appear small. Each of the three hospitals presented chose to present their outcomes in a different format. While informative, it would be hard for families, payers, or governmental agencies to directly compare the clinical outcomes of these three institutions. Transparency in pediatric outcomes seems to have become an established tradition among many pediatric healthcare organizations. What is now needed is standardization of what outcomes should be reported and how they should be presented to make them easily understandable to patients and families.
KeywordsChildren Hospitals Outcomes Transparency Healthcare
Despite decades of discussion and experimentation concerning publicly reported healthcare outcomes, most patients and their families cannot ascertain the relative performance of the healthcare professionals into whose hands they put their children’s lives. Such asymmetry of knowledge commonly occurs in business transactions such as buying a used car, the seller always having the advantage, because he/she knows more about a specific car than the buyer can possibly learn prior to the sale. However, in an information-saturated world, consumers can learn much more about car dealerships, restaurants, or hotels’ goods and services than they can about hospitals and healthcare providers. Consumers can rarely obtain comparative, clinically important outcomes information about pediatric health care. Such an asymmetry of information does not serve patients and their families nor does it bode well for pediatric healthcare as an industry or as a mission-driven enterprise. Either healthcare professionals will provide comprehensible healthcare information or someone else will. The current knowledge asymmetry appears as an anomaly in our increasingly interconnected world.
Numerous non-medical entities have already acted to fill the void. U.S. News and World Report (USNWR) publishes annually what is probably the most well-known source of comparative pediatric hospital outcomes information . While some hospitals might complain about the emphasis on “reputation score,” USNWR has incorporated many measures that reflect clinical excellence and clinical efforts to produce high-quality outcomes. USNWR transparently publishes its scoring criteria each year, and one can go to their Website and see how each hospital scored on specialty-specific criteria. The Center for Medicare and Medicaid Services makes available selected outcome measures on its Website, “Hospital Compare,” although many of the measures apply mostly to adult, not pediatric, healthcare facilities . States, such as Minnesota and others, also provide comparative data on healthcare , as do commercial sites such as Angie’s List, Consumers’ Reports, Healthgrades.com, and many others. Commercial sites may represent the voice of the consumer but often lack the clinical or analytical expertise to provide families with reliable information on clinical outcomes. This leads to the question of interest, “What information should pediatric hospitals report publicly and how should they report it?” This review seeks to discuss the benefits and risk of outcomes reporting and what several pediatric healthcare organizations are doing to self-report their outcomes, primarily on the Internet.
History of outcomes reporting
Healthcare has a long history of measuring and utilizing clinical outcome information. Semmelweiss , Florence Nightingale , and Ernest Codman  all used comparative clinical outcomes to measure and improve their practice of medicine and surgery over a century ago. And all met with considerable criticism, if not hostility, from their peers and superiors. Semmelweiss lost his job, Nightingale was mocked, and Codman was forced to leave his position at the Massachusetts General Hospital and start his own hospital. Little wonder that these pioneering efforts did not lead to wider outcomes reporting for decades. Hospitals and healthcare and provider organizations have for many years unwittingly or purposefully kept their clinical outcomes private, making it difficult for patients to know what they are purchasing and whether or not they are likely to receive maximum value for their purchase. Although most of these organizations expend great energy and resources to measure, assess, and publish their business outcomes such as revenue and income, they have generally spent much less energy on assessing the quality of the services generating their revenue.
It was not until the late twentieth century that outcomes measurements became publicly and widely available. In 1986, the Healthcare Financing Administration (now known as the Centers for Medicare and Medicaid Services or CMS) published hospital mortality rates. This effort generated an outcry based on methodological flaws, such as inadequate risk adjustment, and met an early termination in 1993. However, other groups began to study common cardiovascular procedural and other clinical outcomes. One of the most successful was the Northern New England Cardiovascular Disease Study Group (NNECDSG), which tracked coronary artery bypass procedure outcomes. The NNECDSG addressed issues of data accuracy, audited participating hospitals, utilized clinical data, and developed risk-adjustment models. Most importantly, their reporting was accompanied by collaborative quality improvement efforts. Other groups such as the Vermont Oxford Network and Veteran’s Administration and states, such as New York, Pennsylvania, Minnesota, and others, have also developed registries of clinical outcomes and to one degree or another made at least their aggregated data public. Professional societies, such as the Society of Thoracic Surgeons , also allow the public to view outcomes of commonly performed procedures by hospital. The process of tracking, analyzing, and reporting outcomes either publicly or confidentially seems well underway.
The Northern New England Cardiovascular Disease Study Group
The Northern New England Cardiovascular Disease Study Group (NNECVDSG) serves as a good example of public reporting . They pioneered public reporting of cardiovascular outcomes and detailed their reports to the level of the surgeon. Their efforts began in 1987 as a collaboration of cardiovascular centers in Maine, New Hampshire, and Vermont . They started with coronary artery bypass grafting (CABG) outcomes and have since expanded to include other cardiovascular procedures. They found considerable variation in CABG outcomes despite risk adjustment. However, they did not limit their work to publishing outcomes; they coupled the reporting with collaborative efforts to identify and implement best practices. The NNECVDSG reduced mortality by an impressive 24 % .
Why healthcare organizations resist publicizing their outcomes
Getting outcomes information into the public domain has been an understandably difficult challenge and due to cogent concerns. A number of rationalizations have been made to explain why transparency cannot be achieved. These include the following:
While much has been published about legal issues regarding reporting of specific medical errors or poor outcomes to individual patients, there appears to be no study of the generalized legal risks to hospitals for reporting overall quality (or lack thereof). Of course, all released data must meet privacy guidelines (no identifiable information, etc.). If that criterion is met, and given that revealing errors to the individuals affected has not increased malpractice claims [11, 12], it appears that hospitals or practices should not be concerned about legal ramifications of transparency. Furthermore, on the face of the matter, it seems unlikely that a member of the public at large could demonstrate legal standing to bring a case against a hospital or practice based on composite data.
Validity of data/risk adjustment
Organizations, especially those who have concerns about how their quality data looks, may be reluctant to report and instead take the attitude that “our patients are different” and that without appropriate risk adjustment, they will be perceived as providing lower quality care when that is not in fact the case. Unfortunately, it is also a statistical truism that half of pediatric healthcare organizations must produce below-average outcomes. Each organization will undoubtedly have a strong urge to find a way to report data that never represents themselves as being in the bottom half. Risk adjustment attempts to validate hospital comparisons by adjusting outcomes for the risk factors/differences in the patient population at the beginning of treatment. This remains an imperfect effort. For example, adjusting mortality estimates for admittance through the emergency department might seem reasonable because these patients might be sicker. However, being admitted through the emergency department may have different implications at different hospitals . Furthermore, there may be intrinsic differences in patient populations that cannot be captured by currently known risk factors. Additionally, clinicians have failed to agree on the clinical outcomes of greatest value to patients and families. Therefore, each hospital chooses which outcomes it will report, many of which may not be reported by other hospitals. Goff et al.  recently reported on the attitudes of QI leaders on publicly reported CMS measures. The vast majority of comments addressed perceived invalidity, unimportance, or unfairness of the reported measures. These results mirror multiple other publications in other contexts.
Inability of public to comprehend data
Hospitals may have concerns about the public’s ability to understand more generalized data regarding quality of care provided. A white paper from the Joint Commission  examined whether consumers could correctly interpret data that compares hospital performance. Test subjects reported that they wanted as much information as possible, but actually providing an abundance of data led to more difficulty with comprehension. The authors tested 24 prototypes of data presentation and found that consumers could generally interpret data, but they were unable to find one method of data display which was universally superior to others. One consistently difficult display was a run chart with multiple hospitals, and one consistent improvement to understanding was to display any graphs with “up” being “good.”
Questions about how the public will use the data (or not)
As discussed in the next section, one may question whether the public will do anything with publicly available quality data. Faber et al.  performed a review of the literature and conclude that the ability of public reporting to influence consumer behavior will be limited until a few barriers are overcome which include making the public aware that such data is available, helping the public understand presented data, changing attitudes about leaving a physician or health system with poorer quality of care, and removing financial/geographic/logistical barriers to seeking the highest quality providers.
Benefits of transparency to individual hospitals
A few questions face the institution which is considering increasing its transparency. First, the institution must make a strategic decision as to whether they will be more transparent about costs, quality, or both (e.g., value). Second, the institution must decide whether to be transparent to specific organizations alone (e.g., payers and regulatory agencies) or whether to become openly transparent to everyone including prospective patients, the community, and competitors. Finally, the answers to these questions may depend on the hospital’s aims, and their assessment of how transparency drives those aims. Potentially, the hospital may suspect that transparency to all may affect market share, may motivate internal changes to increase quality and/or costs, or may affect goodwill (the amount of value a company’s reputation adds to overall value).
Improvement in quality of care
Numerous studies have shown an association between public reporting and improved clinical outcomes [17, 18, 19]. However, not all studies have been conclusive and a recent Cochrane Review gave only a tepid endorsement of public reporting as a method to improve outcomes . Despite some reservations, public reporting has repeatedly been associated with improved outcomes, especially in cardiovascular procedures. However, transparency can also distort quality improvement efforts, with hospitals working more to improve the measure than the quality of care . Upcoding to inflate the expected mortality, refusal to treat high-risk patients, and increased racial disparities have all been occasionally associated with public outcomes reporting [22, 23, 24]. Confidential reporting of outcomes to physicians and organizations may be as effective as public reporting . Public or confidential outcomes reporting will only improve care if it motivates clinicians and healthcare organizations to improve care processes. A recent survey of hospital staff suggested that public outcomes reporting provided a needed impetus for organizations to improve or maintain quality . Indeed, the key function of public outcomes reporting may be to stimulate healthcare organizations to improve the quality of their care, which is reflected in improved outcomes over time. It is difficult to improve what one cannot measure, and competitive measurements may be particularly valuable in improving performance.
One other study of whether transparency might lead to improvement in the quality of care provided was published by Lindenauer . Their study examined facilities which publicly reported quality of care metrics, either with or without pay-for-performance incentives funded by CMS. They found that adding a pay-for-performance incentive led to greater improvement in the quality of care as compared to public reporting alone; however, hospitals without pay-for-performance did show improvement over time, and they were not able to compare the group with public reporting to hospitals with no public reporting. Finally, Hibbard et al.  examined the effect of public reporting of hospital quality of care in a specific geographic area. They found that the odds of improvement in quality of care was higher in hospitals with public reporting than hospitals which received non-public report cards, which was greater than hospitals which received no report card (these last hospitals showed no improvement at all). While there were no market shifts with public reporting, the public was able to identify top- and bottom-performing hospitals because of the wide reporting of the quality metrics in the media.
If one suspects that transparency of quality metrics leads to improved market share, the presumptive mechanism is through competition against hospitals with lower (perceived or reported) quality. However, public outcomes reporting appears to have far less effect on market share than it does on efforts to improve clinical outcomes. Healthcare organizations might worry that if their publicly reported outcomes are inferior to others in their service area, they would lose market share. To date, this concern lacks a strong evidence base.
An excellent study on the question of competition and quality choice, which touches on both the questions of market share and hospital performance, has recently been published by Lewis and Pflum . These authors note that evidence of quality report cards driving consumer choice is limited (also reported by Dranove and Sfekas ). They find that competition (which can only take place in the context of transparency to those making a choice) for HMO or MCO patients leads to higher quality of care for all patients in the hospital, while competition for Medicare patients leads to lower quality for all patients. They suggest that this difference arises because of the relative importance placed on quality versus cost by these two segments of the market.
In California and New York, after public reporting, high-mortality, high-complication hospitals experienced only modest transient or no changes in market share or volume . One study reported that one, well-publicized, unexpected death had far more effect on market share than publicly reported hospital mortality data . Furthermore, hospitals’ performance in a specific region may differ little, so that outcomes data appear so similar as to be unhelpful in distinguishing hospitals . These examples notwithstanding, the public’s appetite for comparative hospital performance data may be changing. The current generation of new parents have grown up in the information age of smartphones and wide availability of comparative rankings of everything from vacation spots to vacuum cleaners . It would seem reasonable that although outcomes data have not influenced market share in the past, it may well do so in the future.
Three children’s hospitals’ approaches to transparency
Children’s Hospitals and Clinics of Minnesota
Cincinnati Children’s Hospital Medical Center
The vision of Cincinnati Children’s Hospital Medical Center (CCHMC) is to be the leader in improving child health, and our mission is to achieve the demonstrably best medical and quality of life outcomes, patient and family experience, and value. As a key driver of healthcare transformation to achieve our vision and mission, Cincinnati Children’s developed a transparency strategy in 2006 based on three main premises: (1) Transparency strengthens a culture of inquiry and learning by providing all levels of the organization access to the same performance data and empowering employees to act upon it and improve; (2) sharing information externally promotes an open and collaborative healthcare environment and invites others beyond our geographic area to partner and explore new opportunities to improve child health; and (3) making data available meets and exceeds the expectations of families and helps support their healthcare decisions for their children.
Our approach to incorporate transparency into work of all strategic initiatives led us to a standardized approach to all of our publically reported data. First, we assure that all of the outcomes we report are relevant to families and healthcare providers; the data is checked for accuracy and is accompanied by operational definitions so that other healthcare organizations can replicate the measures if desired. The posted outcomes are accompanied by important patient, family, and/or workforce stories to engage the audience. Finally, each transparently reported measure tells the reader of ongoing process improvement work. Most measures are updated monthly or quarterly based on data availability.
As a result of our dedication to transparency, Cincinnati Children’s was the recipient of the 2012 Hospital Website Transparency award given by URAC and the Leapfrog Group. CCHMC was one of only two hospitals to receive distinction with honors in this area.
Nationwide Children’s Hospital
Conclusion: where do we go from here?
Despite considerable progress in making clinical outcomes available for pediatric healthcare, significant barriers need to be addressed. First, what outcomes are most important to patients and their families? Rarely do hospitals report measures of patient-centered care, rather they report clinical outcomes. The voice of the patient needs to be more prominent in the selection of outcomes measures. Currently, clinical measures lack standardization. Clinicians and administrators usually determine which outcomes will be reported to the public. The variety of outcomes reported and the almost universal favorable standing of the reporting hospital in those reported outcomes suggest that hospitals select outcomes to be reported, in part, based on whether or not they show the hospital in a favorable light. Furthermore, there are over a hundred potential measures endorsed by the National Quality Forum . Which of those best represent clinical quality of child health? Finally, how would a family know which measures should most concern them? Standardization, preferably by a group with clinical expertise and political clout, seems the next most important step to helping our families know the value of the care that they provide for their children. We hope that clinical leaders will take the lead on establishing standard outcomes for public reporting.
Compliance with Ethics Guidelines
Conflict of Interest
Thomas Bartman declares that he has no conflict of interest.
Bobbie Carroll declares that she has no conflict of interest.
Evaline Alessandrini declares that she has no conflict of interest.
Nathaniel R. Payne declares that he has no conflict of interest.
Human and Animal Rights and Informed Consent
This article does not contain any studies with human or animal subjects performed by any of the authors.
References and Recommended Reading
- 1.U.S. news and world report on the best pediatric hospitals. http://health.usnews.com/best-hospitals/pediatric-rankings. Accessed 17 Aug 2015.
- 2.Hospital Compare Website. https://www.medicare.gov/hospitalcompare/search.html. Accessed 17 Aug 2015.
- 3.Minnesota community measurement. http://mncm.org/reports-and-websites/reports-and-data/. Accessed 17 Aug 2015.
- 7.Society of Thoracic Surgeons’ Website. http://www.sts.org/adult-public-reporting-module-search-CABG-hospital?field_npi_name_value=&year_filter=2&state_filter=NY. Accessed 17 Aug 2015.
- 8.Northern New England Cardiovascular Disease Study Group. http://www.nnecdsg.org/data_cabgallsites4.htm. Accessed 17 Aug 2015.
- 10.O’Connor GT, Plume SK, Olmstead EM, Morton JR, Maloney CT, Nugent WC, et al. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. JAMA. 1996;275(11):841–6.CrossRefPubMedGoogle Scholar
- 11.Boothman RC, Imhoff SJ, Campbell DA. Nurturing a culture of patient safety and achieving lower malpractice risk through disclosure: lessons learned and future directions. Front Health Serv Manag. 2012;28(3):13–28.Google Scholar
- 12.Stewart RM, Corneille MG, Johnston J, Geoghegan K, Myers JG, Dent DL, et al. Transparent and open discussion of errors does not increase malpractice risk in trauma patients. Ann Surg. 2006;243(5):645–9. doi: 10.1097/01.sla.0000217304.65877.27. discussion 9-51.PubMedCentralCrossRefPubMedGoogle Scholar
- 14.Goff SL, Lagu T, Pekow PS, Hannon NS, Hinchey KL, Jackowitz TA, et al. A qualitative analysis of hospital leaders’ opinions about publicly reported measures of health care quality. Jt Comm J Qual Patient Saf / J Comm Resources. 2015;41(4):169–76.Google Scholar
- 15.J H, S S. Exploring consumer understanding and use of electronic hospital quality information. In: Commission J, editor.2012.Google Scholar
- 28.Lewis MS, Pflum KE. Competition and quality choice in hospital markets. In: University C, editor.2015.Google Scholar
- 32.Safavi KC, Dai F, Gilbertsen TA, Schonberger RB. Variation in surgical quality measure adherence within hospital referral regions: do publicly reported surgical quality measures distinguish among hospitals that patients are likely to compare? Health Serv Res. 2014;49(4):1108–20. doi: 10.1111/1475-6773.12164.PubMedCentralCrossRefPubMedGoogle Scholar
- 35.National Quality Forum. http://www.qualityforum.org/Measures_Reports_Tools.aspx. Accessed Aug 24 2015.