Misspecification issues in risk adjustment and construction of outcome-based quality indicators

Abstract

Hospital “report cards” reporting risk-adjusted health outcomes are increasingly used to benchmark quality of care. However, risk adjustment methods that do not fully account for the interrelationship between quality, risks and outcomes may lead to biased quality measures. This study aims to determine whether the current approach based on logistic regression and observed-to-expected outcome comparisons (O−E difference or O/E ratio) provides unbiased measures of quality. We first provided a conceptual framework to demonstrate that O−E difference or O/E ratio is inconsistently specified when estimates are based on logistic risk adjustment models. To examine the misspecification issue empirically, risk adjustment was performed based on coronary artery bypass graft (CABG) surgery data from New York’s Cardiac Surgery Reporting System, and quality indicators (QI) of different specifications were calculated for hospital profiling. Computer simulations further explored the issue of misspecified QIs. Results showed that risk-adjusted mortality rates (RAMR) calculated from different QIs identified the same hospital outliers based on 95% confidence intervals, but generated different rank orders for hospitals in both high-quality and low-quality tails of the quality distributions. Simulation results further showed that, compared to O−E and O/E, logistically transformed QIs were superior regarding their abilities to identify hospitals of true extreme rankings, especially when the outcome was less prevalent or the number of patients per hospital was small. Based on our findings, we recommend that analysts consider the use of logistically transformed QI prior to publicly releasing quality rankings using measures based on O−E or O/E.

Appendix

1.1 Appendix 1: Proof of consistent quality indicator specifications

The O−E difference is consistent with the additive linear probability function below:

$$ \begin{aligned} O_{ij}&=x_{ij} \beta^{\prime}+Q_j +\varepsilon _{ij},\\ &=E_{ij}+Q_j+\varepsilon_{ij}\\ \end{aligned} $$

(4)

in which x _ij represents the set of observed risk factors of the patient, β stands for the effect of risks on patient outcome, Q _j is hospital J ’s quality of care that interacts in an additive fashion with patient’s observed risks and can be estimated as either fixed- or random-effects; ${\varepsilon _{ij}}$ represents the random (chance) effect of unknown factors on patient outcome, and ${E_{ij} =x_{ij} \beta'}$ is patient’s predicted outcome rate. We assume that the expectation (average) of the chance effect over all patients in hospital j equals zero.

That is,

$${\bf EXPECT}(\varepsilon _{ij})=0$$

(5)

If we rearrange Eq. 4 and take expectation over hospital j on both sides, we obtain

$$ \begin{aligned} {\bf EXPECT}(\bar{O}_j-\bar{E}_j)&={\bf EXPECT}\,\left[{\frac{1}{n_j }\sum\limits_{i=1}^{n_j}{(O_{ij})}-\frac{1}{n_j}\sum\limits_{i=1}^{n_j }{(E_{ij})}}\right]\\ &={\bf EXPECT}\left[ {\frac{1}{n_j}\sum\limits_{i=1}^{n_j}{(O_{ij}-E_{ij})}}\right] \\ &={\bf EXPECT}\left[ {\frac{1}{n_j}\sum\limits_{i=1}^{n_j}{(Q_{j}+\varepsilon_{ij})}} \right] \\ &={\bf EXPECT}\left[ {\frac{1}{n_j}\sum\limits_{i=1}^{n_j} {(Q_{j})}} \right]\\ &={\bf EXPECT}(Q_j)\\ &=Q_j^d \\ \end{aligned} $$

because of the assumption in Eq. 5, where Q ^d _j is hospital j’s risk-adjusted QI assuming the difference specification.

Similarly, the O/E ratio is consistent with the linear probability function below:

$$ \begin{aligned} O_{ij}&=x_{ij} \beta'\times Q_j +\varepsilon _{ij},\\ &=E_{ij}\times Q_j+\varepsilon _{ij}\\ \end{aligned} $$

(6)

in which hospital quality Q _j interacts multiplicatively with a patient’s observed risks. If we assume that Eq. 5 also holds here and take expectation over hospital j in Eq. 6, we obtain

$$ \begin{aligned} {\bf EXPECT}(\bar{O}_j /\bar {E}_j)&={\bf EXPECT}\left[ {{\sum\limits_{i=1}^{n_j}{(O_{ij})}}\mathord{\left/ {\vphantom {{\sum\limits_{i=1}^{n_j}{(O_{ij})}}{\sum\limits_{i=1}^{n_j}{(E_{ij} )}}}}\right.\kern-\nulldelimiterspace} {\sum\limits_{i=1}^{n_j}{(E_{ij} )}}}\right]\\ &={\bf EXPECT}\left[ {{\sum\limits_{i=1}^{n_j}{(E_{ij}\times Q_j +\varepsilon _{ij})}} \mathord{\left/{\vphantom {{\sum\limits_{i=1}^{n_j}{(E_{ij}\times Q_j +\varepsilon_{ij})}}{\sum\limits_{i=1}^{n_j}{(E_{ij} )}}}}\right. \kern-\nulldelimiterspace}{\sum\limits_{i=1}^{n_j}{(E_{ij})}}}\right] \\ &={\bf EXPECT}\left[ {{\sum\limits_{i=1}^{n_j}{(E_{ij}\times Q_j )}}\mathord{\left/ {\vphantom {{\sum\limits_{i=1}^{n_j}{(E_{ij}\times Q_j )}} {\sum\limits_{i=1}^{n_j}{(E_{ij})}}}}\right.\kern-\nulldelimiterspace} {\sum\limits_{i=1}^{n_j}{(E_{ij})}}}\right]\\ &={\bf EXPECT}\left[ {{Q_j \times \sum\limits_{i=1}^{n_j}{(E_{ij})}}\mathord{\left/ {\vphantom{{Q_j \times \sum\limits_{i=1}^{n_j}{(E_{ij})}} {\sum\limits_{i=1}^{n_j}{(E_{ij})}}}}\right.\kern-\nulldelimiterspace} {\sum\limits_{i=1}^{n_j}{(E_{ij})}}}\right]\\ &={\bf EXPECT}(Q_j)\\ &=Q_j^r \\ \end{aligned} $$

because of the assumption in Eq. 5, where Q ^r _j is hospital j’s risk-adjusted QI measured by the ratio specification.

1.2 Appendix 2

Before simulating the additive logistic specification (Eq. 2), we estimated additive fixed-effects logistic regression on the empirical CABG data:

$$ {\bf logit}(p_{ij})=x_{ij}\beta'\,+(q_2+q_3+\cdots+q_{36} )+\varepsilon_{ij} $$

in which x _ij is the set of risk factors (Table 3), β is the coefficient to be estimated. The fixed-effects q _j were estimated as the coefficient of dummy variables for hospitals (hospital 1 was omitted, q ₁ = 0). Robust variance estimates (White 1980) were obtained to account for the clustering of patients in hospitals.

After model estimation, we calculated the expected mortality as

$$ {\bf logit}(E_{ij})=(x_{ij}\hat{\beta }')+\sum\limits_{j=1}^{36}{\left({\frac{n_j}{N}\times q_j} \right)}, $$

in which n _j is the number of patients in hospital j, N = 16,120 is the total sample size. Therefore, a patient’s expected outcome was calculated as a linear function of patient risk (${x_{ij} \hat{\beta}^{\prime})}$ plus the weighted-average quality effect assuming that the patient were treated in an average quality hospital in New York. Finally, consistent QI (Q ^ld _j ) was constructed. Estimates from this model were then used for simulating the additive Eq. 2.

Before simulating the multiplicative Eq. 3, we estimated multiplicative fixed-effects logistic regression on the empirical data:

$$ {\bf logit}(p_{ij})=x_{ij}\beta'\times(1+q_2 +q_3+\cdots +q_{36})+\varepsilon _{ij}. $$

For the model to be identified, we normalized the effect of hospital 1 to unit. Maximum likelihood estimation in STATA (Gould and Sribney 1999) was used to obtain robust variance estimates.

The expected mortality was calculated as

$$ {\bf logit}(E_{ij})=(x_{ij} \hat{\beta}')\times\sum\limits_{j=1}^{36} {\,\left({\frac{n_j}{N}\times q_j}\right)} $$

in which the weighted-average quality effect takes a multiplicative form on patient risks. Finally, consistent QI (Q ^lr _j ) was constructed. Estimates based on this model were used for subsequent simulations.

  The STATA code for defining the likelihood function of the multiplicative model is listed below: