Optimal Bayesian probability levels for hospital report cards

Abstract

There is a growing trend towards the production of “hospital report-cards” in which hospitals with higher than acceptable mortality rates are identified. Several commentators have advocated for the use of Bayesian hierarchical models in provider profiling. These methods are frequently based upon the posterior probability that a hospital’s mortality rate exceeds a specific benchmark. However, the minimum probability level required for classifying a hospital as having higher than acceptable mortality has never been formally justified. We developed Bayes Rules for determining optimal probability levels so as to minimize mean posterior costs associated with false classifications under specific loss functions. Using Monte Carlo simulation methods we then determined the ability of posterior tail probabilities of unacceptable performance to accurately identify hospitals with higher than acceptable mortality.

Appendix A. Schematic description of Monte Carlo simulations

• 1. For each hospital, draw a random log-odds of mortality and a random slope from the following bivariate normal distribution:

  $$ {\left( \begin{aligned} & \beta _{{0j}} \\ & \beta _{{1j}} \\ \end{aligned} \right)}\,\sim\,{\text{MVN}}{\left( {{\left( {\begin{array}{*{20}l} {{ - 2.063} \hfill} \\ {{0.913} \hfill} \\ \end{array} } \right)},{\left( {\begin{array}{*{20}c} {{0.081}} & {{0.022}} \\ {{0.022}} & {{0.023}} \\ \end{array} } \right)}} \right)},\quad j = 1, \ldots ,109. $$

• 2. If exp (β_0j /β ₀) > 1.25 (where β₀ = −2.063) then the jth hospital is defined to have higher than acceptable mortality (this is the gold standard). The analysis will then be repeated with thresholds of 1.3, 1.4 and 1.5.

• 3. For each hospital, draw a hospital-specific mean illness severity score: $\mu _{j}\,\sim\,N(0,{\text{SD}} = 0.159),\quad j = 1, \ldots ,110.$

• 4. For each patient at the jth hospital, randomly generate an illness severity score: $X_{{ij}}\,\sim\,N(\mu _{j} ,{\text{SD}} = 0.989),\quad i = 1, \ldots ,n_{j} .$

• 5. For each patient, determine the patient-specific log-odds of death: $\log {\left( {\frac{{p_{{ij}} }} {{1 - p_{{ij}} }}} \right)} = \beta _{{0j}} + \beta _{{1j}} X_{{ij}} .$

• 6. Randomly generate each patient’s vital status: $Y_{{ij}}\,\sim\,{\text{Bernouilli}}(p_{{ij}} ).$

• 7. We have now randomly generated an illness severity score and an outcome for each patient at each of the 109 hospitals.

• 8. We now fit a Bayesian hierarchical model to the randomly generated data.

• 9. Using the simulated data, each hospital is classified as having either acceptable mortality or unacceptably high mortality.

• 10. Sensitivity and specificity are determined.

• 11. Steps 1 through 10 are repeated 100 times.

• 12. The mean sensitivity and specificity is determined over the 100 simulations.