Sensitivity Analyses for Unmeasured Confounders

Abstract

Purpose of Review

This review expands on sensitivity analyses for unmeasured confounding techniques, demonstrating state-of-the-art methods as well as specifying which should be used under various scenarios, depending on the information about a potential unmeasured confounder available to the researcher.

Recent Findings

Methods to assess how sensitive an observed estimate is to unmeasured confounding have been developed for decades. Recent advancements have allowed for the incorporation of measured confounders in these assessments, updating the methods used to quantify the impact of an unmeasured confounder, whether specified in terms of the magnitude of the effect from a regression standpoint, for example, as a risk ratio, or with respect to the percent of variation in the outcome or exposure explained by the unmeasured confounder. Additionally, single number summaries, such as the E-value or robustness value, have been proposed to allow for ease of computation when less is known about a specific potential unmeasured confounder.

Summary

This paper aimed to provide methods and tools to implement sensitivity to unmeasured confounder analyses appropriate for various research settings depending on what is known or assumed about a potential unmeasured confounder. We have provided mathematical justification, recommendations, as well as R code to ease the implementation of these methods.

Introduction

An untestable but crucial assumption when trying to determine whether some exposure causes an outcome is the “no unmeasured confounders” assumption [1, 2]. This requires that all factors that cause both the exposure and outcome are measured and adjusted for as part of the analysis. While there is not a test that can assess whether this assumption has been met, sensitivity analyses can be conducted to establish how sensitive your analysis is to some specified unmeasured confounder. There are several flavors of sensitivity analyses that help assess this; the choice of which to use often depends on factors such as the outcome model used to estimate your effect and how confident you are in your estimation of the unmeasured confounder’s relationship with the exposure and outcome. Determining which type of sensitivity analysis best fits your needs is a function of three components:

• (1) What (if anything) is known about the unmeasured confounder-exposure relationship?

• (2) What (if anything) is known about the unmeasured confounder-outcome relationship?

• (3) What model do you intend to use to estimate the relationship between the exposure and outcome?

We construct scenarios under various combinations of the above components to demonstrate the best tools to answer the question at hand given the information available. We provide R code to conduct each sensitivity analysis. The methods described in this paper can be applied to new research as well as previously conducted studies. We hope to arm researchers with the tools needed to select and implement the appropriate sensitivity analysis for the scenario they find themselves in.

Mathematical Notation

This paper attempts to avoid too much mathematical jargon; however, for simplicity, we refer to the exposure as X, the outcome as Y, the unmeasured confounder as U, and all measured confounders as $\mathbf {Z}$. We use the following notation to denote an effect from a conditional model:

$$\begin{aligned} {outcome}\sim {factor of interest} | {additional factors included in the model} \end{aligned}$$

For example, the exposure effect from a model that includes any measured confounders and the unmeasured confounder would be denoted as $Y\sim X | U + \mathbf {Z}$; the exposure effect from a model that includes only measured confounders would be denoted as $Y\sim X | \mathbf {Z}$. Effects can be represented as coefficients ($\beta$), risk ratios (RR), odds ratios (OR), or hazard ratios (HR). Table 1 provides a summary of the notation used going forward.

Table 1 Mathematical Notation

Overview

Sensitivity analyses for unmeasured confounders rely on:

• (1) The observed exposure-outcome effect (after adjusting for all measured confounders, if need be)

• (2) The estimated relationship between an unmeasured confounder and the exposure

• (3) The estimated relationship between an unmeasured confounder and the outcome

The choice of sensitivity analyses depends on whether (2) and (3) are known and with what level of confidence. In scenarios where the researcher is not aware of a specific unmeasured confounder (or does not know the relationship between a specific unmeasured confounder and the exposure and/or outcome) additional assumptions can be overlaid to allow for a broad sensitivity analysis to be applied.

The Observed Outcome Effect

The outcome model choice will depend on the distribution of the outcome, Y, as well as the target effect of interest. If Y is continuous, a linear model may be fit to estimate the effect between X and Y, either via ordinary least squares or a generalized linear model (GLM) with a Gaussian distribution and identity link. Here, the effect of interest is often a $\beta$ coefficient. If Y is binary, the effect of X on Y can be estimated using a GLM with a binomial distribution and log link, a Poisson distribution and log link, or a binomial distribution and logit link. In all models, the $\beta$ coefficient for the exposure, X, will give an estimate of the effect of interest. Alternatively, if a risk ratio is of interest, this can be obtained by exponentiating the $\beta$ coefficient from the GLM with a binomial distribution and log link or the GLM with a Poisson distribution and log link. Note, if estimating the effect on a binary outcome using a GLM with the Poisson distribution and log link function, it is important to use a sandwich estimator to appropriately estimate the variability [3]. An odds ratio can be obtained by exponentiating the $\beta$ coefficient from the GLM with a binomial distribution and logit link. If Y is a time to event outcome, a Cox proportional hazards model is appropriate. The hazard ratio is the exponentiated $\beta$ coefficient for X from the proportional hazards model. Table 2 summarizes the mapping of outcome type and effect of interest to model choice.

Table 2 Mapping of outcome and effect of interest to model choice and effect

The Unmeasured Confounder-Exposure Effect

If the unmeasured confounder, U, is assumed to be binary, this effect can be summarized with two quantities: the prevalence of U in the exposed group ($p_1$) and the prevalence of U in the unexposed group ($p_0$). If U is assumed to be continuous, as with many sensitivity analyses, for simplicity we need to make a few additional assumptions. We can assume that the U has a Gaussian distribution with a mean of $m_1$ in the exposed group, $m_0$ in the unexposed group, and unit variance (a variance of 1). Any normally distributed confounder can fit this specification by scaling by the standard deviation. The only quantity we will need to specify for our sensitivity analysis is the difference in means ($d = m_1 - m_0$). Alternatively, we can disregard the distribution of U and describe the relationship with X via the partial $R^2_{X\sim U|\mathbf {Z}}$, that is the proportion of variation in X explained by U after removing the effect of the measured confounders.

The Unmeasured Confounder-Outcome Effect

A standard way to quantify the unmeasured confounder-outcome effect is to estimate what the coefficient for the unmeasured confounder would be in the fully adjusted outcome model, $\beta _{Y \sim U | X + \mathbf {Z}}$. Equivalently, we could estimate the exponentiated coefficient (the risk ratio, odds ratio, or hazard ratio depending on the model fit). Alternatively, in the case of a continuous outcome, we can disregard the distribution of U and describe the relationship with Y via the partial $R^2_{Y\sim U|X+\mathbf {Z}}$, that is the proportion of variation in Y explained by U after removing the effect of the exposure and the measured confounders.

Approximating the Risk Ratio

The sensitivity analyses presented here that involve an exponentiated coefficient assume that you are estimating a risk ratio. Simulations have shown that these same tools can be applied to odds ratios and hazard ratios when overall event rate is low or high (<10% or >90%) or the size of the unmeasured confounder-outcome effect is relatively small ($|\beta _{Y \sim U | X + \mathbf {Z}}| < 0.75$) [4•]. In the case of a common outcome, simple transformations can be applied to the hazard ratios and odds ratios to better approximate a risk ratio [5]. These approximations work best if the event rate is >20% (equivalently, <80%, as the event can be reverse coded). For event rates between 10 and 20% the approximations will be conservative, since the transformation brings the estimate closer to the null, which is generally preferable for a sensitivity analysis. Note that from this point forward, we will refer to the exponentiated coefficients as a risk ratio; the odds ratio or hazard ratio can be substituted in using the transformations (if needed) as shown in Table 3.

Table 3 Risk ratio approximation

Scenario 1: Sensitivity Analysis for a Particular Unmeasured Confounder

In this first scenario, the researcher has a particular unmeasured confounder in mind, where the relationship between the unmeasured confounder and exposure, U and X, as well as the relationship between the unmeasured confounder and outcome, U and Y, is well understood. For this, we can use methods that have origins that date back to the 1950s when the relationship between smoking and lung cancer was being established using observational data [4•, 6,7,8,9,10]. We can quantify the impact by specifying the assumed unmeasured confounder-outcome and unmeasured confounder-exposure relationships via a simple algebraic equation.

If the outcome is binary or the time to an event (fit via one of the methods outlines in Table 2), we can either update the coefficient for the exposure from the model ($\beta _{Y\sim X | U + \mathbf {Z}}$) or the exponentiated coefficient (the risk ratio or odds ratio for a binary outcome or hazard ratio for a time to event outcome). Here, U is assumed to be independent of the observed confounders, $\mathbf {Z}$, conditional on the exposure, and can be Normally distributed or binary. If U is assumed to be Normally distributed, it has a mean of $m_0$ in the unexposed group, $m_1$ in the exposed group, unit variance, a difference in means of d, and an association with Y of ${RR}_{Y \sim U | X + \mathbf {Z}}$ (or $\beta _{Y \sim U | X + \mathbf {Z}}$ on the linear scale). Since this Normally distributed unmeasured confounder has been standardized to have unit variance, $\beta _{Y \sim U | X + \mathbf {Z}}$ is the standardized regression coefficient. Binary U are assumed to have a prevalence of $p_0$ in the unexposed group, $p_1$ in the exposed group, and an association with Y of ${RR}_{Y \sim U | X + \mathbf {Z}}$ (or $\beta _{Y \sim U | X + \mathbf {Z}}$ on the linear scale). Note that the risk ratio can be replaced with the odds ratio or hazard ratio when the outcome is rare, or the transformation described in Table 3 can be used otherwise. Under the assumption that the sensitivity parameters are fixed, the variance of the observed effect is the same as the variance of the adjusted effect. This allows all adjustments to apply to confidence intervals the same way they would apply to point estimates. Note that the assumption of independence between the unmeasured confounder and observed confounders conditional on the exposure is conservative; that is, if the unmeasured confounder were not independent as assumed, the actual impact would be less than is estimated by these methods.

When the outcome is continuous and Y is related to X, $\mathbf {Z}$, and U through a linear model, we can estimate the updated coefficient, $\beta _{Y\sim X | U + \mathbf {Z}}$, after adjusting for U and the known measured confounders using the same parameterization as above [11, 12]. If the conditional effect of the exposure on the unmeasured confounder is known, we can relax the distributional and independence assumptions, describing the relationship between the unmeasured confounder and the exposure as $\beta _{U\sim X | \mathbf {Z}}$, that is the effect of the exposure on the unmeasured confounder after adjusting for the measured confounders. Alternatively, rather than parameterizing the sensitivity analysis with respect to the coefficients, prevalences, or differences in the means, we could parameterize it with respect to partial $R^2$, the proportion of variation in the exposure explained by the unmeasured confounder and the proportion of variance in the outcome explained by the unmeasured confounder [13•]. We can parameterize the strength of an unmeasured confounder using $R^2$ such that the relationship between U and X is described by $R^2_{X\sim U|\mathbf {Z}}$, the partial $R^2$ of the confounder the exposure, and the relationship between U and Y is described by $R^2_{Y\sim U |X+\mathbf {Z}}$, the partial $R^2$ of the confounder with the outcome. We can then update the observed coefficient, $\beta _{Y\sim X | \mathbf {Z}}$ in the presence of the unmeasured confounder ($\beta _{Y\sim X|U+\mathbf {Z}}$). Again, this method has the advantage of being agnostic to the distribution of the unmeasured confounder.

These sensitivity analyses can be conducted in R using the tipr package [14, 15]. Table 4 shows the equations and R function needed to estimate the effect after adjusting for an unmeasured confounder under each outcome-unmeasured confounder combination.

Table 4 Sensitivity analysis equations and R code

The tipr R Package

The tipr package can be installed by running the following.

install.packages("tipr")

The functions have shared grammar. The function names follow this form: {action}_{effect}_{with what}. For example, to adjust (action) a hazard ratio (effect) with a binary unmeasured confounder (with what), we use the function adjust_hr_with_binary(). There are shorthand aliases for these functions where the “default” assumes the confounder type is continuous, therefore adjust_coef() is equivalent to adjust_coef_with_continuous().

The functions intended to adjust for a normally distributed unmeasured confounder with unit variance have the following arguments:

• effect_observed: The observed exposure-outcome relationship. This can be the point estimate, lower confidence bound, or upper confidence bound.

• exposure_confounder_effect: The estimated difference in means between the unmeasured confounder in the exposed population and unexposed population

• confounder_outcome_effect The estimated relationship between the unmeasured confounder and the outcome

The functions intended to adjust for a binary unmeasured confounder have the following arguments:

• effect_observed: The observed exposure-outcome relationship. This can be the point estimate, lower confidence bound, or upper confidence bound.

• exposed_confounder_prev: The estimated prevalence of the unmeasured confounder in the exposed population.

• unexposed_confounder_prev: The estimated prevalence of the unmeasured confounder in the unexposed population.

• confounder_outcome_effect The estimated relationship between the unmeasured confounder and the outcome

The functions intended to adjust an effect with partial $R^2$ values specified are built on the sensemakr package [16] and have the following arguments:

• effect_observed: The observed exposure-outcome relationship. This is the point estimate.

• se: The standard error for the observed exposure-outcome relationship.

• df: The residual degrees of freedom from the model used to fit the observed exposure-outcome relationship. This is the total number of observations minus the number of parameters estimated in your model. Often for models estimated with an intercept this is $N - k - 1$ where k is the number of predictors in the model.

• confounder_exposure_r2: The estimated partial $R^2$ of the unobserved confounder with the exposure given the measured confounders.

• confounder_outcome_r2 The estimated partial $R^2$ of the unobserved confounder with the outcome given the exposure and the measured confounders.

Functions where the {effect} is an odds ratio (or) or a hazard ratio (hr) include a parameter that allows the user to specify whether they would like the transformations described in Table 3 applied, or_correction and hr_correction, respectively. The default for this parameter is FALSE.

The output for all functions is a data frame with the adjusted effect in the first column, the observed effect in the second column, and the specified sensitivity parameters in the subsequent columns. For example, if we want to know the impact of an unmeasured confounder with $d = 0.5$ and $\beta _{Y \sim U|X +\mathbf {Z}} = 0.25$ on an observed effect of $\beta _{Y \sim X|\mathbf {Z}} = 0.3$, we would run the following code; the resulting output is in Table 5.

library(tipr)

adjust_coef(effect_observed = 0.3,

exposure_confounder_effect = 0.5,

confounder_outcome_effect = 0.25)

Table 5 Example tipr output

Example

To demonstrate this scenario, we will use an analysis that examined the relationship between the choice of two diabetes drugs, metformin or sulfonylurea, and cancer incidence in a cohort of veterans in the United States [17]. This study included a propensity matched cohort of 42,217 patients that were new metformin monotherapy users and 42,217 patients that were new sulfonylureas monotherapy users. Of note, previous studies attempting to answer this question had been fraught with designs suffering from time-related biases, thus this new user cohort was designed specifically to avoid those biases. Even with this careful design, due to the observational nature of the study, there is the potential for unmeasured confounding, and therefore sensitivity analyses were necessitated. An adjusted Cox proportional hazards regression model was fit to examine the relationship between metformin use and the incidence of lung renal cancer when compared to sulfonylurea use. The propensity score model included comorbidities, indicators of health care utilization, demographics, clinical and laboratory values, medications, location of care, and indicators for missingness. The outcome models adjusted for the same set of covariates. In an approach similar to the intention-to-treat analysis used in clinical trials, subsequent changes to the medication regimen were ignored, making persistent exposure not required.

The adjusted hazard ratio for the association with lung cancer in metformin users compared to sulfonylureas users was 0.87 (95% CI 0.79, 0.96). In this scenario, we are concerned about an unmeasured confounder with a known relationship with both the exposure and outcome. In particular, suppose we are concerned that we were unable to adjust for alcohol consumption. Several studies have shown a potential increased risk of lung cancer among heavy drinkers [18]. The 2015 Health Related Behaviors Survey reported that more than 5% of military personnel are heavy drinkers [19]. If there is differential alcohol consumption such that heavy drinking is more prevalent in the sulfonlyureas users, the observed result could be overstating the true effect. What would be a plausible level of differential under consumption? Let’s assume that heavy drinking is prevalent in 4% of the metformin users and 6% in sulfonylureas users, and the relationship between heavy alcohol consumption and lung cancer after adjusting for the other confounders is a hazard ratio of 2. We can plug these quantities into the equation in Table 4 to estimate how this would impact our effect. Let’s suppose we are interested in updating the upper bound of the confidence interval, as this is the bound closest to the null, for this alcohol consumption confounder.

• ${HR}_{Y \sim X |\mathbf {Z}} = 0.96$

• ${HR}_{Y \sim U | X + \mathbf {Z}} = 2$

• $p_1 = 0.04$

• $p_0 = 0.06$

$$\begin{aligned} {HR}_{Y \sim X | U + \mathbf {Z}} \approx 0.96\times \frac{2 \times 0.06+(1-0.06)}{2 \times 0.04+(1-0.04)} \end{aligned}$$

In this study, the event rate was low (<2%), so this approximation will hold. This results in an adjusted hazard ratio of 0.98 for the upper bound. If we plug in the point estimate, 0.87, and the lower bound, 0.79, into the equation, we see an overall adjusted effect of 0.89 (95% CI: 0.81–0.98). Should an unmeasured confounder like this exist, our effect of metformin on lung cancer incidence would be attenuated and fall much closer to the null (with an upper bound of 0.98).

Rather than plugging these values into the equation manually, we can use the adjust_hr_with_binary() function in the tipr package to adjust the hazard ratios (the lower bound, point estimate, and upper bound of the observed effect) for the unmeasured confounder as specified. The output is a table (Table 6) with the effect adjusted for the unmeasured confounder in the first column (${HR}_{Y \sim X | U + \mathbf {Z}}$), and the values specified in the subsequent columns.

library(tipr)

adjust_hr_with_binary(effect_observed = c(0.79, 0.87, 0.96),

exposed_confounder_prev = 0.04,

unexposed_confounder_prev = 0.06,

confounder_outcome_effect = 2)

Table 6 Output from the tipr package showing the impact of a potential known confounder on the effect of metformin (compared to sulfonylurea) on lung cancer diagnosis

Scenario 2: Only one of Unmeasured Confounder-Exposure or Unmeasured Confounder-Outcome Relationships is Known

In this second scenario, the researcher understands the relationship between either the unmeasured confounder and the exposure or the unmeasured confounder and the outcome but not both. We will present two types of sensitivity analyses: an array-based approach that examines the impact of range of sensitivity parameters for the unknown relationship, similar to that presented by Schneeweiss (2006), and a tipping point sensitivity analysis.

Array-Based Sensitivity Analysis

Using the same methods outlined in Section 4, we can plug in the sensitivity parameter(s) for the known relationship and examine a range of sensitivity parameters for the unknown relationship. We could then examine how this range of parameters updates the effect of interest in a table or figure. For example, if we wanted to examine the impact of a Normally distributed unmeasured confounder with unit variance and an assumed difference in means between exposure groups of $d = 0.5$, on an observed odds ratio of 1.3, we could examine a range of unmeasured confounder-outcome effects (for example, from 1 to 2 by 0.1) (Fig. 1).

library(ggplot2)

adjust_df <- adjust_or(

effect_observed = 1.3,

exposure_confounder_effect = 0.5,

confounder_outcome_effect = seq(1, 2, by = 0.1))

ggplot(adjust_df,

aes(x = confounder_outcome_effect,

y = or_adjusted)) +

geom_hline(yintercept = 1.3, lty = 2) +

geom_point() +

geom_line() +

labs(x = "Unmeasured confounder - outcome effect (OR)",

y = "Adjusted OR")

Fig. 1

Impact of a Normally distributed unmeasured confounder with an assumed difference in means between exposure groups of 0.5 on an observed odds ratio of 1.3 (dashed line). The x-axis shows the assumed relationship between the unmeasured confounder and outcome. They y-axis shows the corresponding relationship between the exposure and outcome after adjusting for the unmeasured confounder

Or, if we wanted to examine the impact of a binary unmeasured confounder with an assumed unmeasured confounder-outcome effect of 1.5 on an observed odds ratio of 1.3, we could examine a range of prevalences (for example, from 0 to 1 by 0.2) (Fig. 2).

adjust_df <- adjust_or_with_binary(

effect_observed = 1.3,

exposed_confounder_prev = rep(seq(0, 1, by = 0.2), each = 6),

unexposed_confounder_prev = rep(seq(0, 1, by = 0.2), times = 6),

confounder_outcome_effect = 1.5

)

ggplot(adjust_df,

aes(x = unexposed_confounder_prev,

y = or_adjusted,

group = exposed_confounder_prev)) +

geom_hline(yintercept = 1.3, lty = 2) +

geom_point() +

geom_line() +

geom_label(

data = adjust_df[1:6,],

aes(x = unexposed_confounder_prev,

y = or_adjusted,

label = exposed_confounder_prev)) +

labs(x = "Prevelance of the unmeasured confounder in the unexposed group",

y = "Adjusted OR")

Fig. 2

Impact of a binary unmeasured confounder with an assumed relationship with the outcome of (OR = 1.5) on an observed odds ratio of 1.3 (dashed line). The x-axis shows the assumed prevalence of the unmeasured confounder in the unexposed group; each line represents a different prevalence of the unmeasured confounder in the exposed group. The y-axis shows the corresponding relationship between the exposure and outcome after adjusting for the unmeasured confounder

Tipping Point Sensitivity Analysis

The main objective of a tipping point sensitivity analysis is to report the qualities of an unmeasured confounder needed to bring your observed effect to a particular value, often the null. Rather than looking at a range of values for the unknown sensitivity parameter, we can find the value that would “tip” the observed effect. For example, suppose we observed a risk ratio of 1.25 with a 95% confidence interval (1.1, 1.5); should adjusting for a hypothetical unmeasured confounder cause the lower bound to cross 1, the confidence interval would include the null, and thus the observed effect would be “tipped.” Alternatively, a less conservative approach would examine the hypothetical unmeasured confounder that would cause the point estimate to cross 1. The “tipping point” analysis would find the smallest possible effect of an unmeasured confounder that would cause this to happen. We can do this by setting the adjusted outcome to the null and rearrange the equations in Table 4 to be a function of a single sensitivity parameter, given the remaining parameters. For example, if we were estimating a risk ratio and we knew the relationship between the unmeasured confounder and the exposure (d, the difference in the unmeasured confounder’s means between the exposure groups), we could estimate the unmeasured confounder-outcome effect (${RR}_{Y \sim U | X + \mathbf {Z}}$) needed to “tip” the observed effect to the null — that is, the value that would make ${RR}_{Y \sim X |U+ \mathbf {Z}} = 1$ — as shown in Eq. (1).

$$\begin{aligned} {RR}_{Y \sim U | X + \mathbf {Z}, {tip} = 1} = {RR}_{Y \sim X | \mathbf {Z}}^{1/d} \end{aligned}$$

(1)

Here, ${RR}_{Y \sim U | X + \mathbf {Z}, {tip} = 1}$ is a function of the observed effect, ${RR}_{Y \sim X | \mathbf {Z}}$, and the assumed unmeasured confounder-exposure relationship, d, assuming that the “tipping point” for the effect after adjusting for the unmeasured confounder (${RR}_{Y \sim X | U + \mathbf {Z}}$) is 1. Table 7 shows the equations for all combinations of tipping point scenarios as well as the R function from the tipr package, depending on the effect of interest, unmeasured confounder type, and known unmeasured confounder relationship. For simplicity, we have only included the risk ratio; however, this can be exchanged for the odds ratio or hazard ratio according to the transformations described in Table 3. Notice that if the unmeasured confounder-outcome relationship is known and the unmeasured confounder is assumed to be binary, at least one of the prevalences of the unmeasured confounder ($p_0$ or $p_1$) must be specified. While the sensitivity analyses that assume the binary or Normally distributed unmeasured confounders can be applied to the point estimate, lower bound, or upper bound, the $R^2$ parameterization, as written, only applies to the point estimate. The tipr package allows you to tip at the bound rather than the point estimate by setting the tip_bound = TRUE option in the tip_coef_with_r2() function.

Table 7 Tipping point equations and R code

Example

To demonstrate these methods, we will use a study that examined the impact of an evidence-based ICU intervention (known as the ABCDEF bundle) on a patient’s likelihood of mechanical ventilation [20]. This study included 10,840 patients who had at least two consecutive 24-h ICU days recruited from 68 adult academic, community and Veterans Administration ICUs from 29 states and Puerto Rico. A logistic regression model was fit with robust sandwich estimation, clustered by study site, to examine the relationship between ABCDEF bundle use and mechanical ventilation, adjusting for demographic variables, admission features, and daily ICU characteristics on the day of bundle exposure. This resulted in an adjusted odds ratio of 0.28 (95% CI, 0.22–0.36), indicating that a patient with complete ABCDEF bundle performance on a given day also had a significantly lower likelihood of mechanical ventilation.

A key missing confounder was the severity of illness. Only six of the 68 sites collected an indicator for severity of illness (APACHE III), and therefore it was not possible to adjust for in the main analysis. Because these six sites may not have been representative of the remaining 62 (they were nearly all academic institutions, whereas over 1/3 of the 68 were community hospitals), the relationship between severity of illness and the outcome (mechanical ventilation) in the overall cohort was unknown. These six institutions were, however, able to provide an assumed difference in means of the severity of illness score (APACHE III) between the exposure groups. The observed difference in means between exposure groups was 0.41 after scaling by the standard deviation. We can use this in a tipping point sensitivity analysis to examine the effect between severity of illness score and mechanical ventilation that would be needed to tip the observed odds ratio of 0.28 (95% CI, 0.22–0.36). Because this odds ratio is less than 1, the limiting bound (the bound closest to the null), is the upper bound. We can plug this into the tip_or() function from the tipr package. Because this outcome is not rare, we will use the or_correction parameter to apply the square-root transformation referenced in Table 3.

tip_or(effect_observed = 0.36,

exposure_confounder_effect = 0.41,

or_correction = TRUE)

After applying the transformation, our upper bound for the approximate risk ratio is 0.6. The tipping point analysis reveals that in order to tip this analysis to include the null, the risk ratio for the association between severity of illness and mechanical ventilation would need to be 0.28. This would be a large and unlikely effect. This allowed the researchers to conclude that while ABCDEF bundle effect may not be as large as was observed, there is still likely to be a clinically relevant association.

Scenario 3: Nothing is Known About a Potential Unmeasured Confounder

In this final scenario, nothing is known about a potential unmeasured confounder. In this case, three methods can be used: an array-based approach, similar to that described in Section 5.1, an approach that uses the measured confounders to ground the analysis (such as the approach by Cinelli and Hazlett (2020) which parameterizes sensitivity analyses using $R^2$), or a single number summary such as the robust value [13•] or E-value [21•]. The final two approaches do not require any specification of sensitivity parameters, making them particularly appealing when nothing is known about a particular unmeasured confounder.

Array-Based Sensitivity Analysis

Similar to Section 5.1, if nothing is known about the unmeasured confounder, an array-based approach can be used to explore the potential space. One way to do this would be to expand the plots fit in Section 5.1. For example, if we wanted to examine the impact of a Normally distributed unmeasured confounder on an observed odds ratio of 1.3, we could examine a range of unmeasured confounder-outcome effects (for example, from 1.2 to 2 by 0.2) as well as a range of differences in means between exposure groups (for example, from 0.2 to 1 by 0.2) (Fig. 3).

library(ggplot2)

adjust_df <- adjust_or(

effect_observed = 1.3,

exposure_confounder_effect = rep(seq(0.2, 1, by = 0.2), times = 5),

confounder_outcome_effect = rep(seq(1.2, 2, by = 0.2), each = 5))

ggplot(adjust_df,

aes(x = confounder_outcome_effect,

y = or_adjusted,

group = exposure_confounder_effect)) +

geom_hline(yintercept = 1.3, lty = 2) +

geom_point() +

geom_line() +

geom_label(

data = adjust_df[21:25,],

aes(x = confounder_outcome_effect,

y = or_adjusted,

label = exposure_confounder_effect)) +

labs(x = "Unmeasured confounder - outcome effect (OR)",

y = "Adjusted OR")

Fig. 3

Impact of a Normally distributed unmeasured confounder on an observed odds ratio of 1.3 (dashed line). The x-axis shows the assumed relationship between the unmeasured confounder and outcome and the lines show the varying difference in means between the exposure groups. The y-axis shows the corresponding relationship between the exposure and outcome after adjusting for the unmeasured confounder

Similarly, if we wanted to examine the impact of a binary unmeasured confounder, we could either examine a 3-dimensional plot to look across the 3 sensitivity parameters, or we could fix one and examine a range of the remaining 2. For example, we could fix the prevalence of the unmeasured confounder in the exposure group to 1 (as this will be maximally conservative) and examine a range of prevalence in the unexposed group as well as a range of unmeasured confounder-outcome effects.

Another alternative would be to combine the tipping point sensitivity analysis and the array-based approach, examining a range of the relationship between the unmeasured confounder and either the exposure or outcome and finding the tipping point for the other. This is akin to a “bias plot” or “sensitivity contour plots” [13•, 22]. For example, if we wanted to examine the impact of a binary unmeasured confounder on an observed risk ratio of 2, we could conservatively set the prevalence in the exposed population to 1, vary the prevalence in the unexposed population (for example, from 0.05 to 0.4 by 0.05) and find the unmeasured confounder-outcome relationship that would tip this (Fig. 4).

tip_df <- tip_with_binary(

effect_observed = 2,

exposed_confounder_prev = 1,

unexposed_confounder_prev = seq(0.05, 0.4, by = 0.05)

)

ggplot(tip_df,

aes(x = 1/unexposed_confounder_prev,

y = confounder_outcome_effect)) +

geom_point() +

geom_line() +

labs(x = "1 / Prevelance of the unmeasured confounder in the unexposed group",

y = "Unmeasured confounder - outcome effect (RR)")

Fig. 4

Tipping point of a binary unmeasured confounder on an observed risk ratio of 2. The x-axis shows 1 / prevalence of the unmeasured confounder in the unexposed group (the risk ratio of the unmeasured confounder). The prevalence in the exposed group is assumed to be 1. The y-axis shows the assumed relationship between the unmeasured confounder and outcome (RR) that would tip the analysis at the given prevalence

Grounding in the Measured Confounders

When nothing is known about the potential unmeasured confounder, one option is to use the measured confounders to group the sensitivity analysis. This has an advantage over the “array” approach, as it may give some context to the plausibility of a particular confounder. The partial $R^2$ parameterization is a nice way to implement this for continuous outcomes. We can examine the partial $R^2$ for particular covariates we have measured and see how a new unmeasured confounder with same relationship with the exposure and outcome would change the observed result. The partial $R^2$ can conveniently be calculated for each measured confounder using the observed t-statistic and degrees of freedom.

$$\begin{aligned} R^2_{ {partial}}=\frac{t^2}{t^2 + {df}} \end{aligned}$$

The user would estimate this for each measured confounder of interest for both the measured confounder-exposure relationship and measured confounder-outcome relationship. Using the methods described in [13•], these observed relationships can be used to calculate bounds on a partial $R^2$ of an unobserved confounder that is k times as strong as the observed confounder. In R, we can use the ovb_partial_r2_bound function from the sensemakr package to calculate these bounds [16].

We can use the tip_coef_with_r2() function from the tipr package to generate tipping point partial $R^2$ values for the unmeasured confounder-outcome relationship for a range of partial $R^2$ values for the unmeasured confounder-exposure relationship, and plot these values, akin to the plots in the previous section. We can then add points for observed partial $R^2$ values for observed confounders. For example, suppose we had an observed exposure-outcome effect, $\beta _{Y \sim X | \mathbf {Z}}$ of 0.5, with a standard error of 0.1 and 100 degrees of freedom. We accounted for two observed, confounders with the following partial $R^2$ (Fig. 5):

• $Z_1$: $R^2_{X\sim Z_1 | Z_2} = 0.3$, $R^2_{Y\sim Z_1 | X + Z_2} = 0.005$

• $Z_2$: $R^2_{X\sim Z_2 | Z_1} = 0.3$, $R^2_{Y\sim Z_2 | X + Z_1} = 0.15$

tipping_r2 <- tip_coef_with_r2(

effect_observed = 0.5,

se = 0.1,

df = 100,

confounder_outcome_r2 = seq(0, 1, by = 0.1))

## Calculate benchmarks

z1_benchmark <- sensemakr::ovb_partial_r2_bound(0.3, 0.005)

z1_2x_benchmark <- sensemakr::ovb_partial_r2_bound(0.3, 0.005, kd=2)

z2_benchmark <- sensemakr::ovb_partial_r2_bound(0.3, 0.15)

ggplot(tipping_r2,

aes(x = confounder_exposure_r2,

y = confounder_outcome_r2)) +

geom_point() +

geom_line() +

annotate("label",

x = z1_benchmark[["r2dz.x"]],

y = z1_benchmark [["r2yz.dx"]],

label = "Z1") +

annotate("label",

x = z1_2x_benchmark[["r2dz.x"]],

y = z1_2x_benchmark[["r2yz.dx"]],

label = "2x Z1") +

annotate("label",

x = z2_benchmark[["r2dz.x"]],

y = z2_benchmark[["r2yz.dx"]],

label = "Z2") +

labs(x = "partial R2 of the unobserved confounder with the exposure",

y = "partial R2 of the unobserved confounder with the outcome")

Fig. 5

Tipping point of an unmeasured confounder on an observed coefficient of 0.5, a standard error of 0.1 and 100 degrees of freedom. The x-axis represents the partial $R^2$ value of the unobserved confounder with the exposure and the y-axis represents the partial $R^2$ value of the unobserved confounder with the outcome. The line shows values that would tip the observed effect. Two benchmark observed covariates are shown, $Z_1$ and $Z_2$ An unobserved confounder as strong as $Z_1$ would not tip the analysis, however an unobserved confounder as 2x strong as $Z_1$ (denoted by 2x $Z_1$) would tip the analysis, as evidenced by this point falling to the right of the threshold line. Additionally, an unobserved confounder as strong as $Z_2$ would tip the analysis as well

Single Number Summaries

Robustness Value

An additional benefit of this $R^2$ paramaterization is it allows for a single number summary, the robustness value, defined as follows (Eq. (2)) [13•].

$$\begin{aligned} RV = \frac{1}{2}\left\{ \sqrt{f^4+4f^2}-f^2\right\} \end{aligned}$$

(2)

Where f is the partial Cohen’s f of the exposure with the outcome, or the exposure coefficient’s t-value divided by $\sqrt{ {df}}$. Similarly, this can be applied to the confidence bounds by replacing f with $f - f^*_{\alpha , {df}-1}$ in the above equation, where $f^*_{\alpha , {df}-1}$ is the t-value threshold for a t-test with $\alpha$-level significance and df-1 degrees of freedom divided by $\sqrt{ {df}-1}$.

$$\begin{aligned} f^*_{\alpha , {df}-1}=|t^*_{\alpha , {df}-1}|/\sqrt{ {df}-1} \end{aligned}$$

This allows the researcher to state that unobserved confounders that explain RV% of the residual variability in both the exposure and outcome are sufficiently strong to explain away the observed effect. This is similar to the tipping point analysis present above, except instead of specifying one relationship and finding a tipping point for the other, the two are jointly minimized. The r_value() function in the tipr package will calculate this.

The E-Value

Ding and VanderWeele (2016) suggest focusing on the point that minimizes the strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both the exposure and outcome, conditional on the measured covariates, to explain away an observed exposure-outcome association [21•, 22]. They call this value an “E-value” (Eq. (3)).

$$\begin{aligned} {E-value} = {RR}_{Y\sim X | \mathbf {Z}} + \sqrt{ {RR}_{Y\sim X | \mathbf {Z}}\times ( {RR}_{Y\sim X | \mathbf {Z}}-1)} \end{aligned}$$

(3)

Note if your observed effect is less than 1, you can take the inverse and apply Eq. (3). In the binary case, this is equivalent to the tipping point analysis where $p_1$ is set to 1 and $p_0$ and ${RR}_{Y\sim U | X + \mathbf {Z}}$ are selected such that $1/p_0$ and ${RR}_{Y\sim U | X + \mathbf {Z}}$ are equal (the minimum values needed to tip the analysis). This is equivalent to finding the point closest to the bottom left corner in Fig. 4. The E-value method adds simplification, in that no sensitivity parameters need to be specified; however, it may not generalize well. This may result in an ambiguous number, as it is not intrinsically grounded in the observed covariates and does not take into account plausible associations [23,24,25,26,27]. For example, this bounding factor may be unnecessarily conservative in many settings where a prevalence of 1 in the exposed population is not plausible.

Using the tipr R package, if we wanted to know the E-value for a risk ratio of 2, we could use the e_value() function as follows. This would give an E-value of 3.41.

Other Methods

We want to emphasize that there has been extensive research in this area; We were not able to explore the extensive methods in the Bayesian space. Greenland (2001) describes a Bayesian approach to sensitivity to unmeasured confounders analyses using Monte Carlo risk assessment. Greenland (1998) explains that the common method for approaching a sensitivity analysis, treating the unmeasured confounders as fixed values as if they are known, does not formally incorporate the uncertainty about the sensitivity parameters and can be sensitive to the specification of the unmeasured confounder. He demonstrates that under certain circumstances, the output from a Monte Carlo risk adjustment with priors for the sensitivity parameters can approximate the posterior that would be obtained from a Bayesian analysis. Greenland (2003) further describes choosing priors for bias parameters and demonstrates how even in the case of a relatively low prior probability that an unmeasured confounder explains the association between an exposure and outcome, introducing unmeasured confounders in this manner can considerably increase the uncertainty of a causal relationship. These results are further summarized and described as Monte Carlo sensitivity analyses (MCSA) by Greenland (2005). Similarly, McCandless, Gustafson, and Levy (2007) describe Bayesian sensitivity analyses for unmeasured confounding using MCMC. They build on methods put forth by Lin, Psaty, and Kronmal (1998) to build Bayesian models with prior distributions used for the sensitivity analyses that approximate the sampling distribution of model parameters in a hypothetical sequence of observational studies. They demonstrate that credible intervals will on average have approximately nominal coverage probability under these circumstances. The authors further show that sensitivity analyses using information about measured confounders can improve the determination of the uncertainty of unmeasured confounders [28]. They assert that if the confounding effect of the unmeasured confounder is similar to that of the measured confounders, the Bayesian Sensitivity Analysis may give results that overstate the uncertainty about bias.

Discussion

Any analysis that makes the “no unmeasured confounders” assumption ought to include a sensitivity analysis to assess the potential impact of this assumption, particularly those for which unmeasured confounding is likely. While imperfect, the inclusion of sensitivity analyses in some form is an important step to begin to address this issue. The methods presented here hopefully provide enough variety of plausible scenarios as well as practical advice to make their implementation straightforward. This paper is most useful for the researcher who is concerned about the presence of an unmeasured confounder, but might not know the exact relationship of this confounder with the measured covariates, exposure, and outcome, as well as the uncertainty involved. If all quantities were known, one could backward engineer an unmeasured confounder that has specified prevalences in each exposure and a given association with the outcome while not changing your existing dataset’s outcome, exposure, and covariates. This simulation would answer a slightly different question, which is how would the confidence intervals shift, probabilistically, if the unmeasured confounder were measured with some uncertainty. This is slightly different from our analysis, which is testing what if the effect of the unmeasured confounder was perfectly known and adjusted for. The latter allows for a simpler description of its impact.

This paper aimed to provide methods and tools to implement sensitivity to unmeasured confounder analyses appropriate for various research settings depending on what is known or assumed about a potential unmeasured confounder. We have provided mathematical justification, recommendations, as well as R code to ease the implementation of these methods.