Abstract
Purpose of Review
This review expands on sensitivity analyses for unmeasured confounding techniques, demonstrating stateoftheart methods as well as specifying which should be used under various scenarios, depending on the information about a potential unmeasured confounder available to the researcher.
Recent Findings
Methods to assess how sensitive an observed estimate is to unmeasured confounding have been developed for decades. Recent advancements have allowed for the incorporation of measured confounders in these assessments, updating the methods used to quantify the impact of an unmeasured confounder, whether specified in terms of the magnitude of the effect from a regression standpoint, for example, as a risk ratio, or with respect to the percent of variation in the outcome or exposure explained by the unmeasured confounder. Additionally, single number summaries, such as the Evalue or robustness value, have been proposed to allow for ease of computation when less is known about a specific potential unmeasured confounder.
Summary
This paper aimed to provide methods and tools to implement sensitivity to unmeasured confounder analyses appropriate for various research settings depending on what is known or assumed about a potential unmeasured confounder. We have provided mathematical justification, recommendations, as well as R code to ease the implementation of these methods.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
An untestable but crucial assumption when trying to determine whether some exposure causes an outcome is the “no unmeasured confounders” assumption [1, 2]. This requires that all factors that cause both the exposure and outcome are measured and adjusted for as part of the analysis. While there is not a test that can assess whether this assumption has been met, sensitivity analyses can be conducted to establish how sensitive your analysis is to some specified unmeasured confounder. There are several flavors of sensitivity analyses that help assess this; the choice of which to use often depends on factors such as the outcome model used to estimate your effect and how confident you are in your estimation of the unmeasured confounder’s relationship with the exposure and outcome. Determining which type of sensitivity analysis best fits your needs is a function of three components:

(1)
What (if anything) is known about the unmeasured confounderexposure relationship?

(2)
What (if anything) is known about the unmeasured confounderoutcome relationship?

(3)
What model do you intend to use to estimate the relationship between the exposure and outcome?
We construct scenarios under various combinations of the above components to demonstrate the best tools to answer the question at hand given the information available. We provide R code to conduct each sensitivity analysis. The methods described in this paper can be applied to new research as well as previously conducted studies. We hope to arm researchers with the tools needed to select and implement the appropriate sensitivity analysis for the scenario they find themselves in.
Mathematical Notation
This paper attempts to avoid too much mathematical jargon; however, for simplicity, we refer to the exposure as X, the outcome as Y, the unmeasured confounder as U, and all measured confounders as \(\mathbf {Z}\). We use the following notation to denote an effect from a conditional model:
For example, the exposure effect from a model that includes any measured confounders and the unmeasured confounder would be denoted as \(Y\sim X  U + \mathbf {Z}\); the exposure effect from a model that includes only measured confounders would be denoted as \(Y\sim X  \mathbf {Z}\). Effects can be represented as coefficients (\(\beta\)), risk ratios (RR), odds ratios (OR), or hazard ratios (HR). Table 1 provides a summary of the notation used going forward.
Overview
Sensitivity analyses for unmeasured confounders rely on:

(1)
The observed exposureoutcome effect (after adjusting for all measured confounders, if need be)

(2)
The estimated relationship between an unmeasured confounder and the exposure

(3)
The estimated relationship between an unmeasured confounder and the outcome
The choice of sensitivity analyses depends on whether (2) and (3) are known and with what level of confidence. In scenarios where the researcher is not aware of a specific unmeasured confounder (or does not know the relationship between a specific unmeasured confounder and the exposure and/or outcome) additional assumptions can be overlaid to allow for a broad sensitivity analysis to be applied.
The Observed Outcome Effect
The outcome model choice will depend on the distribution of the outcome, Y, as well as the target effect of interest. If Y is continuous, a linear model may be fit to estimate the effect between X and Y, either via ordinary least squares or a generalized linear model (GLM) with a Gaussian distribution and identity link. Here, the effect of interest is often a \(\beta\) coefficient. If Y is binary, the effect of X on Y can be estimated using a GLM with a binomial distribution and log link, a Poisson distribution and log link, or a binomial distribution and logit link. In all models, the \(\beta\) coefficient for the exposure, X, will give an estimate of the effect of interest. Alternatively, if a risk ratio is of interest, this can be obtained by exponentiating the \(\beta\) coefficient from the GLM with a binomial distribution and log link or the GLM with a Poisson distribution and log link. Note, if estimating the effect on a binary outcome using a GLM with the Poisson distribution and log link function, it is important to use a sandwich estimator to appropriately estimate the variability [3]. An odds ratio can be obtained by exponentiating the \(\beta\) coefficient from the GLM with a binomial distribution and logit link. If Y is a time to event outcome, a Cox proportional hazards model is appropriate. The hazard ratio is the exponentiated \(\beta\) coefficient for X from the proportional hazards model. Table 2 summarizes the mapping of outcome type and effect of interest to model choice.
The Unmeasured ConfounderExposure Effect
If the unmeasured confounder, U, is assumed to be binary, this effect can be summarized with two quantities: the prevalence of U in the exposed group (\(p_1\)) and the prevalence of U in the unexposed group (\(p_0\)). If U is assumed to be continuous, as with many sensitivity analyses, for simplicity we need to make a few additional assumptions. We can assume that the U has a Gaussian distribution with a mean of \(m_1\) in the exposed group, \(m_0\) in the unexposed group, and unit variance (a variance of 1). Any normally distributed confounder can fit this specification by scaling by the standard deviation. The only quantity we will need to specify for our sensitivity analysis is the difference in means (\(d = m_1  m_0\)). Alternatively, we can disregard the distribution of U and describe the relationship with X via the partial \(R^2_{X\sim U\mathbf {Z}}\), that is the proportion of variation in X explained by U after removing the effect of the measured confounders.
The Unmeasured ConfounderOutcome Effect
A standard way to quantify the unmeasured confounderoutcome effect is to estimate what the coefficient for the unmeasured confounder would be in the fully adjusted outcome model, \(\beta _{Y \sim U  X + \mathbf {Z}}\). Equivalently, we could estimate the exponentiated coefficient (the risk ratio, odds ratio, or hazard ratio depending on the model fit). Alternatively, in the case of a continuous outcome, we can disregard the distribution of U and describe the relationship with Y via the partial \(R^2_{Y\sim UX+\mathbf {Z}}\), that is the proportion of variation in Y explained by U after removing the effect of the exposure and the measured confounders.
Approximating the Risk Ratio
The sensitivity analyses presented here that involve an exponentiated coefficient assume that you are estimating a risk ratio. Simulations have shown that these same tools can be applied to odds ratios and hazard ratios when overall event rate is low or high (<10% or >90%) or the size of the unmeasured confounderoutcome effect is relatively small (\(\beta _{Y \sim U  X + \mathbf {Z}} < 0.75\)) [4•]. In the case of a common outcome, simple transformations can be applied to the hazard ratios and odds ratios to better approximate a risk ratio [5]. These approximations work best if the event rate is >20% (equivalently, <80%, as the event can be reverse coded). For event rates between 10 and 20% the approximations will be conservative, since the transformation brings the estimate closer to the null, which is generally preferable for a sensitivity analysis. Note that from this point forward, we will refer to the exponentiated coefficients as a risk ratio; the odds ratio or hazard ratio can be substituted in using the transformations (if needed) as shown in Table 3.
Scenario 1: Sensitivity Analysis for a Particular Unmeasured Confounder
In this first scenario, the researcher has a particular unmeasured confounder in mind, where the relationship between the unmeasured confounder and exposure, U and X, as well as the relationship between the unmeasured confounder and outcome, U and Y, is well understood. For this, we can use methods that have origins that date back to the 1950s when the relationship between smoking and lung cancer was being established using observational data [4•, 6,7,8,9,10]. We can quantify the impact by specifying the assumed unmeasured confounderoutcome and unmeasured confounderexposure relationships via a simple algebraic equation.
If the outcome is binary or the time to an event (fit via one of the methods outlines in Table 2), we can either update the coefficient for the exposure from the model (\(\beta _{Y\sim X  U + \mathbf {Z}}\)) or the exponentiated coefficient (the risk ratio or odds ratio for a binary outcome or hazard ratio for a time to event outcome). Here, U is assumed to be independent of the observed confounders, \(\mathbf {Z}\), conditional on the exposure, and can be Normally distributed or binary. If U is assumed to be Normally distributed, it has a mean of \(m_0\) in the unexposed group, \(m_1\) in the exposed group, unit variance, a difference in means of d, and an association with Y of \({RR}_{Y \sim U  X + \mathbf {Z}}\) (or \(\beta _{Y \sim U  X + \mathbf {Z}}\) on the linear scale). Since this Normally distributed unmeasured confounder has been standardized to have unit variance, \(\beta _{Y \sim U  X + \mathbf {Z}}\) is the standardized regression coefficient. Binary U are assumed to have a prevalence of \(p_0\) in the unexposed group, \(p_1\) in the exposed group, and an association with Y of \({RR}_{Y \sim U  X + \mathbf {Z}}\) (or \(\beta _{Y \sim U  X + \mathbf {Z}}\) on the linear scale). Note that the risk ratio can be replaced with the odds ratio or hazard ratio when the outcome is rare, or the transformation described in Table 3 can be used otherwise. Under the assumption that the sensitivity parameters are fixed, the variance of the observed effect is the same as the variance of the adjusted effect. This allows all adjustments to apply to confidence intervals the same way they would apply to point estimates. Note that the assumption of independence between the unmeasured confounder and observed confounders conditional on the exposure is conservative; that is, if the unmeasured confounder were not independent as assumed, the actual impact would be less than is estimated by these methods.
When the outcome is continuous and Y is related to X, \(\mathbf {Z}\), and U through a linear model, we can estimate the updated coefficient, \(\beta _{Y\sim X  U + \mathbf {Z}}\), after adjusting for U and the known measured confounders using the same parameterization as above [11, 12]. If the conditional effect of the exposure on the unmeasured confounder is known, we can relax the distributional and independence assumptions, describing the relationship between the unmeasured confounder and the exposure as \(\beta _{U\sim X  \mathbf {Z}}\), that is the effect of the exposure on the unmeasured confounder after adjusting for the measured confounders. Alternatively, rather than parameterizing the sensitivity analysis with respect to the coefficients, prevalences, or differences in the means, we could parameterize it with respect to partial \(R^2\), the proportion of variation in the exposure explained by the unmeasured confounder and the proportion of variance in the outcome explained by the unmeasured confounder [13•]. We can parameterize the strength of an unmeasured confounder using \(R^2\) such that the relationship between U and X is described by \(R^2_{X\sim U\mathbf {Z}}\), the partial \(R^2\) of the confounder the exposure, and the relationship between U and Y is described by \(R^2_{Y\sim U X+\mathbf {Z}}\), the partial \(R^2\) of the confounder with the outcome. We can then update the observed coefficient, \(\beta _{Y\sim X  \mathbf {Z}}\) in the presence of the unmeasured confounder (\(\beta _{Y\sim XU+\mathbf {Z}}\)). Again, this method has the advantage of being agnostic to the distribution of the unmeasured confounder.
These sensitivity analyses can be conducted in R using the tipr package [14, 15]. Table 4 shows the equations and R function needed to estimate the effect after adjusting for an unmeasured confounder under each outcomeunmeasured confounder combination.
The tipr R Package
The tipr package can be installed by running the following.
install.packages("tipr")
The functions have shared grammar. The function names follow this form: {action}_{effect}_{with what}. For example, to adjust (action) a hazard ratio (effect) with a binary unmeasured confounder (with what), we use the function adjust_hr_with_binary(). There are shorthand aliases for these functions where the “default” assumes the confounder type is continuous, therefore adjust_coef() is equivalent to adjust_coef_with_continuous().
The functions intended to adjust for a normally distributed unmeasured confounder with unit variance have the following arguments:

effect_observed: The observed exposureoutcome relationship. This can be the point estimate, lower confidence bound, or upper confidence bound.

exposure_confounder_effect: The estimated difference in means between the unmeasured confounder in the exposed population and unexposed population

confounder_outcome_effect The estimated relationship between the unmeasured confounder and the outcome
The functions intended to adjust for a binary unmeasured confounder have the following arguments:

effect_observed: The observed exposureoutcome relationship. This can be the point estimate, lower confidence bound, or upper confidence bound.

exposed_confounder_prev: The estimated prevalence of the unmeasured confounder in the exposed population.

unexposed_confounder_prev: The estimated prevalence of the unmeasured confounder in the unexposed population.

confounder_outcome_effect The estimated relationship between the unmeasured confounder and the outcome
The functions intended to adjust an effect with partial \(R^2\) values specified are built on the sensemakr package [16] and have the following arguments:

effect_observed: The observed exposureoutcome relationship. This is the point estimate.

se: The standard error for the observed exposureoutcome relationship.

df: The residual degrees of freedom from the model used to fit the observed exposureoutcome relationship. This is the total number of observations minus the number of parameters estimated in your model. Often for models estimated with an intercept this is \(N  k  1\) where k is the number of predictors in the model.

confounder_exposure_r2: The estimated partial \(R^2\) of the unobserved confounder with the exposure given the measured confounders.

confounder_outcome_r2 The estimated partial \(R^2\) of the unobserved confounder with the outcome given the exposure and the measured confounders.
Functions where the {effect} is an odds ratio (or) or a hazard ratio (hr) include a parameter that allows the user to specify whether they would like the transformations described in Table 3 applied, or_correction and hr_correction, respectively. The default for this parameter is FALSE.
The output for all functions is a data frame with the adjusted effect in the first column, the observed effect in the second column, and the specified sensitivity parameters in the subsequent columns. For example, if we want to know the impact of an unmeasured confounder with \(d = 0.5\) and \(\beta _{Y \sim UX +\mathbf {Z}} = 0.25\) on an observed effect of \(\beta _{Y \sim X\mathbf {Z}} = 0.3\), we would run the following code; the resulting output is in Table 5.
library(tipr)
adjust_coef(effect_observed = 0.3,
exposure_confounder_effect = 0.5,
confounder_outcome_effect = 0.25)
Example
To demonstrate this scenario, we will use an analysis that examined the relationship between the choice of two diabetes drugs, metformin or sulfonylurea, and cancer incidence in a cohort of veterans in the United States [17]. This study included a propensity matched cohort of 42,217 patients that were new metformin monotherapy users and 42,217 patients that were new sulfonylureas monotherapy users. Of note, previous studies attempting to answer this question had been fraught with designs suffering from timerelated biases, thus this new user cohort was designed specifically to avoid those biases. Even with this careful design, due to the observational nature of the study, there is the potential for unmeasured confounding, and therefore sensitivity analyses were necessitated. An adjusted Cox proportional hazards regression model was fit to examine the relationship between metformin use and the incidence of lung renal cancer when compared to sulfonylurea use. The propensity score model included comorbidities, indicators of health care utilization, demographics, clinical and laboratory values, medications, location of care, and indicators for missingness. The outcome models adjusted for the same set of covariates. In an approach similar to the intentiontotreat analysis used in clinical trials, subsequent changes to the medication regimen were ignored, making persistent exposure not required.
The adjusted hazard ratio for the association with lung cancer in metformin users compared to sulfonylureas users was 0.87 (95% CI 0.79, 0.96). In this scenario, we are concerned about an unmeasured confounder with a known relationship with both the exposure and outcome. In particular, suppose we are concerned that we were unable to adjust for alcohol consumption. Several studies have shown a potential increased risk of lung cancer among heavy drinkers [18]. The 2015 Health Related Behaviors Survey reported that more than 5% of military personnel are heavy drinkers [19]. If there is differential alcohol consumption such that heavy drinking is more prevalent in the sulfonlyureas users, the observed result could be overstating the true effect. What would be a plausible level of differential under consumption? Let’s assume that heavy drinking is prevalent in 4% of the metformin users and 6% in sulfonylureas users, and the relationship between heavy alcohol consumption and lung cancer after adjusting for the other confounders is a hazard ratio of 2. We can plug these quantities into the equation in Table 4 to estimate how this would impact our effect. Let’s suppose we are interested in updating the upper bound of the confidence interval, as this is the bound closest to the null, for this alcohol consumption confounder.

\({HR}_{Y \sim X \mathbf {Z}} = 0.96\)

\({HR}_{Y \sim U  X + \mathbf {Z}} = 2\)

\(p_1 = 0.04\)

\(p_0 = 0.06\)
In this study, the event rate was low (<2%), so this approximation will hold. This results in an adjusted hazard ratio of 0.98 for the upper bound. If we plug in the point estimate, 0.87, and the lower bound, 0.79, into the equation, we see an overall adjusted effect of 0.89 (95% CI: 0.81–0.98). Should an unmeasured confounder like this exist, our effect of metformin on lung cancer incidence would be attenuated and fall much closer to the null (with an upper bound of 0.98).
Rather than plugging these values into the equation manually, we can use the adjust_hr_with_binary() function in the tipr package to adjust the hazard ratios (the lower bound, point estimate, and upper bound of the observed effect) for the unmeasured confounder as specified. The output is a table (Table 6) with the effect adjusted for the unmeasured confounder in the first column (\({HR}_{Y \sim X  U + \mathbf {Z}}\)), and the values specified in the subsequent columns.
library(tipr)
adjust_hr_with_binary(effect_observed = c(0.79, 0.87, 0.96),
exposed_confounder_prev = 0.04,
unexposed_confounder_prev = 0.06,
confounder_outcome_effect = 2)
Scenario 2: Only one of Unmeasured ConfounderExposure or Unmeasured ConfounderOutcome Relationships is Known
In this second scenario, the researcher understands the relationship between either the unmeasured confounder and the exposure or the unmeasured confounder and the outcome but not both. We will present two types of sensitivity analyses: an arraybased approach that examines the impact of range of sensitivity parameters for the unknown relationship, similar to that presented by Schneeweiss (2006), and a tipping point sensitivity analysis.
ArrayBased Sensitivity Analysis
Using the same methods outlined in Section 4, we can plug in the sensitivity parameter(s) for the known relationship and examine a range of sensitivity parameters for the unknown relationship. We could then examine how this range of parameters updates the effect of interest in a table or figure. For example, if we wanted to examine the impact of a Normally distributed unmeasured confounder with unit variance and an assumed difference in means between exposure groups of \(d = 0.5\), on an observed odds ratio of 1.3, we could examine a range of unmeasured confounderoutcome effects (for example, from 1 to 2 by 0.1) (Fig. 1).
library(ggplot2)
adjust_df < adjust_or(
effect_observed = 1.3,
exposure_confounder_effect = 0.5,
confounder_outcome_effect = seq(1, 2, by = 0.1))
ggplot(adjust_df,
aes(x = confounder_outcome_effect,
y = or_adjusted)) +
geom_hline(yintercept = 1.3, lty = 2) +
geom_point() +
geom_line() +
labs(x = "Unmeasured confounder  outcome effect (OR)",
y = "Adjusted OR")
Or, if we wanted to examine the impact of a binary unmeasured confounder with an assumed unmeasured confounderoutcome effect of 1.5 on an observed odds ratio of 1.3, we could examine a range of prevalences (for example, from 0 to 1 by 0.2) (Fig. 2).
adjust_df < adjust_or_with_binary(
effect_observed = 1.3,
exposed_confounder_prev = rep(seq(0, 1, by = 0.2), each = 6),
unexposed_confounder_prev = rep(seq(0, 1, by = 0.2), times = 6),
confounder_outcome_effect = 1.5
)
ggplot(adjust_df,
aes(x = unexposed_confounder_prev,
y = or_adjusted,
group = exposed_confounder_prev)) +
geom_hline(yintercept = 1.3, lty = 2) +
geom_point() +
geom_line() +
geom_label(
data = adjust_df[1:6,],
aes(x = unexposed_confounder_prev,
y = or_adjusted,
label = exposed_confounder_prev)) +
labs(x = "Prevelance of the unmeasured confounder in the unexposed group",
y = "Adjusted OR")
Tipping Point Sensitivity Analysis
The main objective of a tipping point sensitivity analysis is to report the qualities of an unmeasured confounder needed to bring your observed effect to a particular value, often the null. Rather than looking at a range of values for the unknown sensitivity parameter, we can find the value that would “tip” the observed effect. For example, suppose we observed a risk ratio of 1.25 with a 95% confidence interval (1.1, 1.5); should adjusting for a hypothetical unmeasured confounder cause the lower bound to cross 1, the confidence interval would include the null, and thus the observed effect would be “tipped.” Alternatively, a less conservative approach would examine the hypothetical unmeasured confounder that would cause the point estimate to cross 1. The “tipping point” analysis would find the smallest possible effect of an unmeasured confounder that would cause this to happen. We can do this by setting the adjusted outcome to the null and rearrange the equations in Table 4 to be a function of a single sensitivity parameter, given the remaining parameters. For example, if we were estimating a risk ratio and we knew the relationship between the unmeasured confounder and the exposure (d, the difference in the unmeasured confounder’s means between the exposure groups), we could estimate the unmeasured confounderoutcome effect (\({RR}_{Y \sim U  X + \mathbf {Z}}\)) needed to “tip” the observed effect to the null — that is, the value that would make \({RR}_{Y \sim X U+ \mathbf {Z}} = 1\) — as shown in Eq. (1).
Here, \({RR}_{Y \sim U  X + \mathbf {Z}, {tip} = 1}\) is a function of the observed effect, \({RR}_{Y \sim X  \mathbf {Z}}\), and the assumed unmeasured confounderexposure relationship, d, assuming that the “tipping point” for the effect after adjusting for the unmeasured confounder (\({RR}_{Y \sim X  U + \mathbf {Z}}\)) is 1. Table 7 shows the equations for all combinations of tipping point scenarios as well as the R function from the tipr package, depending on the effect of interest, unmeasured confounder type, and known unmeasured confounder relationship. For simplicity, we have only included the risk ratio; however, this can be exchanged for the odds ratio or hazard ratio according to the transformations described in Table 3. Notice that if the unmeasured confounderoutcome relationship is known and the unmeasured confounder is assumed to be binary, at least one of the prevalences of the unmeasured confounder (\(p_0\) or \(p_1\)) must be specified. While the sensitivity analyses that assume the binary or Normally distributed unmeasured confounders can be applied to the point estimate, lower bound, or upper bound, the \(R^2\) parameterization, as written, only applies to the point estimate. The tipr package allows you to tip at the bound rather than the point estimate by setting the tip_bound = TRUE option in the tip_coef_with_r2() function.
Example
To demonstrate these methods, we will use a study that examined the impact of an evidencebased ICU intervention (known as the ABCDEF bundle) on a patient’s likelihood of mechanical ventilation [20]. This study included 10,840 patients who had at least two consecutive 24h ICU days recruited from 68 adult academic, community and Veterans Administration ICUs from 29 states and Puerto Rico. A logistic regression model was fit with robust sandwich estimation, clustered by study site, to examine the relationship between ABCDEF bundle use and mechanical ventilation, adjusting for demographic variables, admission features, and daily ICU characteristics on the day of bundle exposure. This resulted in an adjusted odds ratio of 0.28 (95% CI, 0.22–0.36), indicating that a patient with complete ABCDEF bundle performance on a given day also had a significantly lower likelihood of mechanical ventilation.
A key missing confounder was the severity of illness. Only six of the 68 sites collected an indicator for severity of illness (APACHE III), and therefore it was not possible to adjust for in the main analysis. Because these six sites may not have been representative of the remaining 62 (they were nearly all academic institutions, whereas over 1/3 of the 68 were community hospitals), the relationship between severity of illness and the outcome (mechanical ventilation) in the overall cohort was unknown. These six institutions were, however, able to provide an assumed difference in means of the severity of illness score (APACHE III) between the exposure groups. The observed difference in means between exposure groups was 0.41 after scaling by the standard deviation. We can use this in a tipping point sensitivity analysis to examine the effect between severity of illness score and mechanical ventilation that would be needed to tip the observed odds ratio of 0.28 (95% CI, 0.22–0.36). Because this odds ratio is less than 1, the limiting bound (the bound closest to the null), is the upper bound. We can plug this into the tip_or() function from the tipr package. Because this outcome is not rare, we will use the or_correction parameter to apply the squareroot transformation referenced in Table 3.
tip_or(effect_observed = 0.36,
exposure_confounder_effect = 0.41,
or_correction = TRUE)
After applying the transformation, our upper bound for the approximate risk ratio is 0.6. The tipping point analysis reveals that in order to tip this analysis to include the null, the risk ratio for the association between severity of illness and mechanical ventilation would need to be 0.28. This would be a large and unlikely effect. This allowed the researchers to conclude that while ABCDEF bundle effect may not be as large as was observed, there is still likely to be a clinically relevant association.
Scenario 3: Nothing is Known About a Potential Unmeasured Confounder
In this final scenario, nothing is known about a potential unmeasured confounder. In this case, three methods can be used: an arraybased approach, similar to that described in Section 5.1, an approach that uses the measured confounders to ground the analysis (such as the approach by Cinelli and Hazlett (2020) which parameterizes sensitivity analyses using \(R^2\)), or a single number summary such as the robust value [13•] or Evalue [21•]. The final two approaches do not require any specification of sensitivity parameters, making them particularly appealing when nothing is known about a particular unmeasured confounder.
ArrayBased Sensitivity Analysis
Similar to Section 5.1, if nothing is known about the unmeasured confounder, an arraybased approach can be used to explore the potential space. One way to do this would be to expand the plots fit in Section 5.1. For example, if we wanted to examine the impact of a Normally distributed unmeasured confounder on an observed odds ratio of 1.3, we could examine a range of unmeasured confounderoutcome effects (for example, from 1.2 to 2 by 0.2) as well as a range of differences in means between exposure groups (for example, from 0.2 to 1 by 0.2) (Fig. 3).
library(ggplot2)
adjust_df < adjust_or(
effect_observed = 1.3,
exposure_confounder_effect = rep(seq(0.2, 1, by = 0.2), times = 5),
confounder_outcome_effect = rep(seq(1.2, 2, by = 0.2), each = 5))
ggplot(adjust_df,
aes(x = confounder_outcome_effect,
y = or_adjusted,
group = exposure_confounder_effect)) +
geom_hline(yintercept = 1.3, lty = 2) +
geom_point() +
geom_line() +
geom_label(
data = adjust_df[21:25,],
aes(x = confounder_outcome_effect,
y = or_adjusted,
label = exposure_confounder_effect)) +
labs(x = "Unmeasured confounder  outcome effect (OR)",
y = "Adjusted OR")
Similarly, if we wanted to examine the impact of a binary unmeasured confounder, we could either examine a 3dimensional plot to look across the 3 sensitivity parameters, or we could fix one and examine a range of the remaining 2. For example, we could fix the prevalence of the unmeasured confounder in the exposure group to 1 (as this will be maximally conservative) and examine a range of prevalence in the unexposed group as well as a range of unmeasured confounderoutcome effects.
Another alternative would be to combine the tipping point sensitivity analysis and the arraybased approach, examining a range of the relationship between the unmeasured confounder and either the exposure or outcome and finding the tipping point for the other. This is akin to a “bias plot” or “sensitivity contour plots” [13•, 22]. For example, if we wanted to examine the impact of a binary unmeasured confounder on an observed risk ratio of 2, we could conservatively set the prevalence in the exposed population to 1, vary the prevalence in the unexposed population (for example, from 0.05 to 0.4 by 0.05) and find the unmeasured confounderoutcome relationship that would tip this (Fig. 4).
tip_df < tip_with_binary(
effect_observed = 2,
exposed_confounder_prev = 1,
unexposed_confounder_prev = seq(0.05, 0.4, by = 0.05)
)
ggplot(tip_df,
aes(x = 1/unexposed_confounder_prev,
y = confounder_outcome_effect)) +
geom_point() +
geom_line() +
labs(x = "1 / Prevelance of the unmeasured confounder in the unexposed group",
y = "Unmeasured confounder  outcome effect (RR)")
Grounding in the Measured Confounders
When nothing is known about the potential unmeasured confounder, one option is to use the measured confounders to group the sensitivity analysis. This has an advantage over the “array” approach, as it may give some context to the plausibility of a particular confounder. The partial \(R^2\) parameterization is a nice way to implement this for continuous outcomes. We can examine the partial \(R^2\) for particular covariates we have measured and see how a new unmeasured confounder with same relationship with the exposure and outcome would change the observed result. The partial \(R^2\) can conveniently be calculated for each measured confounder using the observed tstatistic and degrees of freedom.
The user would estimate this for each measured confounder of interest for both the measured confounderexposure relationship and measured confounderoutcome relationship. Using the methods described in [13•], these observed relationships can be used to calculate bounds on a partial \(R^2\) of an unobserved confounder that is k times as strong as the observed confounder. In R, we can use the ovb_partial_r2_bound function from the sensemakr package to calculate these bounds [16].
We can use the tip_coef_with_r2() function from the tipr package to generate tipping point partial \(R^2\) values for the unmeasured confounderoutcome relationship for a range of partial \(R^2\) values for the unmeasured confounderexposure relationship, and plot these values, akin to the plots in the previous section. We can then add points for observed partial \(R^2\) values for observed confounders. For example, suppose we had an observed exposureoutcome effect, \(\beta _{Y \sim X  \mathbf {Z}}\) of 0.5, with a standard error of 0.1 and 100 degrees of freedom. We accounted for two observed, confounders with the following partial \(R^2\) (Fig. 5):

\(Z_1\): \(R^2_{X\sim Z_1  Z_2} = 0.3\), \(R^2_{Y\sim Z_1  X + Z_2} = 0.005\)

\(Z_2\): \(R^2_{X\sim Z_2  Z_1} = 0.3\), \(R^2_{Y\sim Z_2  X + Z_1} = 0.15\)
tipping_r2 < tip_coef_with_r2(
effect_observed = 0.5,
se = 0.1,
df = 100,
confounder_outcome_r2 = seq(0, 1, by = 0.1))
## Calculate benchmarks
z1_benchmark < sensemakr::ovb_partial_r2_bound(0.3, 0.005)
z1_2x_benchmark < sensemakr::ovb_partial_r2_bound(0.3, 0.005, kd=2)
z2_benchmark < sensemakr::ovb_partial_r2_bound(0.3, 0.15)
ggplot(tipping_r2,
aes(x = confounder_exposure_r2,
y = confounder_outcome_r2)) +
geom_point() +
geom_line() +
annotate("label",
x = z1_benchmark[["r2dz.x"]],
y = z1_benchmark [["r2yz.dx"]],
label = "Z1") +
annotate("label",
x = z1_2x_benchmark[["r2dz.x"]],
y = z1_2x_benchmark[["r2yz.dx"]],
label = "2x Z1") +
annotate("label",
x = z2_benchmark[["r2dz.x"]],
y = z2_benchmark[["r2yz.dx"]],
label = "Z2") +
labs(x = "partial R2 of the unobserved confounder with the exposure",
y = "partial R2 of the unobserved confounder with the outcome")
Single Number Summaries
Robustness Value
An additional benefit of this \(R^2\) paramaterization is it allows for a single number summary, the robustness value, defined as follows (Eq. (2)) [13•].
Where f is the partial Cohen’s f of the exposure with the outcome, or the exposure coefficient’s tvalue divided by \(\sqrt{ {df}}\). Similarly, this can be applied to the confidence bounds by replacing f with \(f  f^*_{\alpha , {df}1}\) in the above equation, where \(f^*_{\alpha , {df}1}\) is the tvalue threshold for a ttest with \(\alpha\)level significance and df1 degrees of freedom divided by \(\sqrt{ {df}1}\).
This allows the researcher to state that unobserved confounders that explain RV% of the residual variability in both the exposure and outcome are sufficiently strong to explain away the observed effect. This is similar to the tipping point analysis present above, except instead of specifying one relationship and finding a tipping point for the other, the two are jointly minimized. The r_value() function in the tipr package will calculate this.
The EValue
Ding and VanderWeele (2016) suggest focusing on the point that minimizes the strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both the exposure and outcome, conditional on the measured covariates, to explain away an observed exposureoutcome association [21•, 22]. They call this value an “Evalue” (Eq. (3)).
Note if your observed effect is less than 1, you can take the inverse and apply Eq. (3). In the binary case, this is equivalent to the tipping point analysis where \(p_1\) is set to 1 and \(p_0\) and \({RR}_{Y\sim U  X + \mathbf {Z}}\) are selected such that \(1/p_0\) and \({RR}_{Y\sim U  X + \mathbf {Z}}\) are equal (the minimum values needed to tip the analysis). This is equivalent to finding the point closest to the bottom left corner in Fig. 4. The Evalue method adds simplification, in that no sensitivity parameters need to be specified; however, it may not generalize well. This may result in an ambiguous number, as it is not intrinsically grounded in the observed covariates and does not take into account plausible associations [23,24,25,26,27]. For example, this bounding factor may be unnecessarily conservative in many settings where a prevalence of 1 in the exposed population is not plausible.
Using the tipr R package, if we wanted to know the Evalue for a risk ratio of 2, we could use the e_value() function as follows. This would give an Evalue of 3.41.
Other Methods
We want to emphasize that there has been extensive research in this area; We were not able to explore the extensive methods in the Bayesian space. Greenland (2001) describes a Bayesian approach to sensitivity to unmeasured confounders analyses using Monte Carlo risk assessment. Greenland (1998) explains that the common method for approaching a sensitivity analysis, treating the unmeasured confounders as fixed values as if they are known, does not formally incorporate the uncertainty about the sensitivity parameters and can be sensitive to the specification of the unmeasured confounder. He demonstrates that under certain circumstances, the output from a Monte Carlo risk adjustment with priors for the sensitivity parameters can approximate the posterior that would be obtained from a Bayesian analysis. Greenland (2003) further describes choosing priors for bias parameters and demonstrates how even in the case of a relatively low prior probability that an unmeasured confounder explains the association between an exposure and outcome, introducing unmeasured confounders in this manner can considerably increase the uncertainty of a causal relationship. These results are further summarized and described as Monte Carlo sensitivity analyses (MCSA) by Greenland (2005). Similarly, McCandless, Gustafson, and Levy (2007) describe Bayesian sensitivity analyses for unmeasured confounding using MCMC. They build on methods put forth by Lin, Psaty, and Kronmal (1998) to build Bayesian models with prior distributions used for the sensitivity analyses that approximate the sampling distribution of model parameters in a hypothetical sequence of observational studies. They demonstrate that credible intervals will on average have approximately nominal coverage probability under these circumstances. The authors further show that sensitivity analyses using information about measured confounders can improve the determination of the uncertainty of unmeasured confounders [28]. They assert that if the confounding effect of the unmeasured confounder is similar to that of the measured confounders, the Bayesian Sensitivity Analysis may give results that overstate the uncertainty about bias.
Discussion
Any analysis that makes the “no unmeasured confounders” assumption ought to include a sensitivity analysis to assess the potential impact of this assumption, particularly those for which unmeasured confounding is likely. While imperfect, the inclusion of sensitivity analyses in some form is an important step to begin to address this issue. The methods presented here hopefully provide enough variety of plausible scenarios as well as practical advice to make their implementation straightforward. This paper is most useful for the researcher who is concerned about the presence of an unmeasured confounder, but might not know the exact relationship of this confounder with the measured covariates, exposure, and outcome, as well as the uncertainty involved. If all quantities were known, one could backward engineer an unmeasured confounder that has specified prevalences in each exposure and a given association with the outcome while not changing your existing dataset’s outcome, exposure, and covariates. This simulation would answer a slightly different question, which is how would the confidence intervals shift, probabilistically, if the unmeasured confounder were measured with some uncertainty. This is slightly different from our analysis, which is testing what if the effect of the unmeasured confounder was perfectly known and adjusted for. The latter allows for a simpler description of its impact.
This paper aimed to provide methods and tools to implement sensitivity to unmeasured confounder analyses appropriate for various research settings depending on what is known or assumed about a potential unmeasured confounder. We have provided mathematical justification, recommendations, as well as R code to ease the implementation of these methods.
Change history
24 January 2023
A Correction to this paper has been published: https://doi.org/10.1007/s40471022003184
References
Papers of particular interest, published recently, have been highlighted as: • Of importance
Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology. American Psychological Association. 1974;66:688.
D’Agostino RB. Propensity score methods for bias reduction in the comparison of a treatment to a nonrandomized control group. Stat Med. 1998;17:2265–81.
Naimi AI, Whitcomb BW. Estimating risk ratios and risk differences using regression. American Journal of Epidemiology. 2020
Lin DY, Psaty BM, Kronmal RA. Assessing the sensitivity of regression results to unmeasured confounders in observational studies. Biometrics. 1998;54:948–63. This paper derives the relationship between a binary or normally distributed unmeasured confounder and a coefficient, relative risk, odds ratio, or hazard ratio, after adjusting for measured covariates.
VanderWeele TJ. Optimal approximate conversions of odds ratios and hazard ratios to risk ratios. Biometrics Wiley Online Library. 2020;76:746–52.
Cornfield J, Haenszel W, Hammond EC, Lilienfeld AM, Shimkin MB, Wynder EL. Smoking and lung cancer: recent evidence and a discussion of some questions. J Natl Cancer Inst. 1959;22:173–203.
Bross I. Spurious effects from an extraneous variable. J Chronic Dis. 1966
Schlesselman JJ. Assessing effects of confounding variables. Am J Epidemiol. 1978;108:3–8.
Rosenbaum PR, Rubin DB. Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society Series B. 1983;
Lash TL, Fox MP, Fink AK, et al. Applying quantitative bias analysis to epidemiologic data. Springer; 2009.
Rosenbaum PR. Dropping out of high school in the united states: An observational study. Journal of Educational Statistics. Sage Publications Sage CA: Los Angeles, CA; 1986;11:207–24.
Draper NR, Smith H. Applied regression analysis. 2nd ed. New York: John Wiley & Sons; 1981.
Cinelli C, Hazlett C. Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society: Series B (Statistical Methodology). Wiley Online Library; 2020;82:39–67. This paper describes how to parameterize sensitivity analyses using \(R^2\)and introduces the robustness value.
McGowan LD. Tipr: Tipping point analyses. 2022. (R package version 1.0.1)
McGowan LD. Tipr: An r package for sensitivity analyses for unmeasured confounders. Journal of Open Source Software. The Open Journal; 2022;7:4495.
Cinelli C, Ferwerda J, Hazlett C. Sensemakr: Sensitivity analysis tools for regression models [Internet]. 2021. Available from: https://CRAN.Rproject.org/package=sensemakr (R package version 0.1.4).
Murff HJ, Roumie CL, Greevy RA, Hackstadt AJ, McGowan LE, Hung AM, et al. Metformin use and incidence cancer risk: Evidence for a selective protective effect against liver cancer. Cancer Causes & Control Springer. 2018;29:823–32.
Bandera EV, Freudenheim JL, Vena JE. Alcohol consumption and lung cancer: A review of the epidemiologic evidence. Cancer Epidemiology Biomarkers & Prevention AACR. 2001;10:813–21.
Meadows SO, Engel CC, Collins RL, Beckman RL, Cefalu M, HawesDawson J, et al. 2015 health related behaviors survey: Substance use among US activeduty service members. RAND; 2018.
Pun BT, Balas MC, BarnesDaly MA, Thompson JL, Aldrich JM, Barr J, et al. Caring for critically ill patients with the ABCDEF bundle: Results of the ICU liberation collaborative in over 15,000 adults. Critical care medicine. NIH Public Access; 2019;47:3.
VanderWeele TJ, Ding P. Sensitivity Analysis in Observational Research: Introducing the EValue. Ann Intern Med. 2017; This paper introduces the Evalue, a single number summary quantifying the joint minimum strength of association with both the exposure and outcome needed to tip an observed effect to cross the null.
Ding P, VanderWeele TJ. Sensitivity Analysis Without Assumptions. Epidemiology (Cambridge, Mass). 2016;27:368–77.
MacLehose RF, Ahern TP, Lash TL, Poole C, Greenland S. The importance of making assumptions in bias analysis. Epidemiology (Cambridge, Mass). Wolters Kluwer Health; 2021;32:617.
Poole C. Commentary: Continuing the evalue’s postpublication peer review. International Journal of Epidemiology. Oxford University Press; 2020;49:1497–500.
Barberio J, Ahern TP, MacLehose RF, Collin LJ, CroninFenton DP, Damkier P, et al. Assessing techniques for quantifying the impact of bias due to an unmeasured confounder: An applied example. Clinical Epidemiology. Dove Press; 2021;13:627.
Greenland S. Commentary: An argument against evalues for assessing the plausibility that an association could be explained away by residual confounding. International Journal of Epidemiology. Oxford University Press; 2020;49:1501–3.
Ioannidis JP, Tan YJ, Blum MR. Limitations and misinterpretations of evalues for sensitivity analyses of observational studies. Annals of internal medicine. American College of Physicians; 2019;170:108–11.
McCandless LC, Gustafson P, Levy AR. A sensitivity analysis using information about measured confounders yielded improved uncertainty assessments for unmeasured confounding. J Clin Epidemiol. 2008;61:247–55.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The author declares no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is part of the Topical Collection on Epidemiologic Methods.
The original online version of this article was revised: The published paper is not updated
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
D’Agostino McGowan, L. Sensitivity Analyses for Unmeasured Confounders. Curr Epidemiol Rep 9, 361–375 (2022). https://doi.org/10.1007/s40471022003086
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40471022003086