Selection Bias

Abstract

Selection bias concerns systematic error that may arise from the manner in which subjects are selected into one’s study. In his lesson, we describe examples of selection bias, provide a quantitative framework for assessing selection bias, show how selection bias can occur in different types of epidemiologic study designs, and discuss how to adjust for or otherwise deal with selection bias.

Appendices

Homework

1.1 ACE-1. Selection Bias: Dietary Fat vs. Colon Cancer

Suppose an investigator is interested in whether dietary fat is associated with the development of colon cancer. S/he identifies several hundred incident cases of colon cancer and a comparable number of community controls. Each subject fills out a food frequency questionnaire describing his/her dietary habits in the previous year; information from the questionnaire is used to divide subjects into those who consumed a high fat diet (exposed) and those who consumed a low fat diet (unexposed).

Assume the following:

• Among cases, the likelihood of agreeing to participate in the study is NOT related to exposure.

• Among controls, those who consume a high fat diet are half as likely to participate, compared to those who consume a low fat diet.

• a. Express each of the assumptions above in terms of selection probabilities.

• b. Calculation the selection odds for the diseased (r_D) and the selection odds for the non-diseased (r_{not D}).

• c. Based on your answers to a and b above, do you expect bias in the estimated odds ratio (OR) from this study. Justify your answer.

Suppose the following table reflects the data that one would have observed if there had been 100 % participation among both cases and controls (i.e., these are unbiased data):

	High Fat	Low Fat	Total
Case	200	175	375
Control	150	225	375
Total	350	400	750

• d. Based on the information about selection probabilities described earlier, describe the 2x2 table that gives the study population for this study.

• e. Compute and compare the odds ratio in the source population (i.e., 100 % participation) with the odds ratio in the study population. Is there bias, and if so, what is the direction of the bias?

1.2 ACE-2. Selection Bias: Food Allergies vs. CHD Development

Consider a ten-year follow-up of a fixed cohort free of CHD at the start of follow-up. The study objective is to determine whether persons experiencing food allergies (FA) prior to the start of follow-up are at increased risk for CHD development. Suppose that a simple analysis of the study results yields the following study information.

	FA
CHD	Yes	No	Total
Yes	49	286	335
No	1378	5816	7194
Total	1427	6102	7529

RR = 0.72, OR = 0.72

Which of the following statements about possible selection bias is appropriate for this study design (Choose only one answer):

• a. If persons who had high blood pressure (a known risk factor for CHD) prior to the start of follow-up were excluded from the comparison (no allergy) group, but NOT from the FA (allergy) group, then there should be concern about selection bias that would be AWAY FROM THE NULL.

• b. If the likelihood of being not lost to follow-up for persons with food allergies is exactly the same as the likelihood of not being lost to follow-up for persons without food allergies, then there can be no selection bias due to follow-up loss.

• c. Bias from selective survival should be a concern in this study.

• d. Berkson’s bias should be a concern in this study.

• e. If persons who were free of any kind of illness (except FA) prior to start of follow-up were excluded from both the exposed group and the comparison (no allergy) group, then there should be concern about selection bias that would be TOWARDS THE NULL.

1.3 ACE-3. Selective Survival

• a. What is meant by “bias due to selective survival” in cross-sectional studies? (In your answer, make sure to define appropriate selection probability parameters.)

• b. Under what circumstances might there be no selective survival bias even if the selection probabilities are not all equal?

• c. Suppose that you could assess that the direction of possible selective survival bias in your study was towards the null. If your study data yielded a non-statistically significant odds ratio of 1.04, would it be correct to conclude that there was no exposure-disease association in your source population? Explain.

1.4 ACE-4. Choice of Cases and Controls

You are asked to advise a clinician interested in conducting two case-control studies examining the relationship between coffee consumption and:

• i. Primary ovarian cancer.

• ii. Coronary artery disease

The clinician wishes to use hospital admissions as the source of cases. Advise how valid the proposal is, particularly in light of possible selection biases. How would you advise him about choice of cases and controls?

1.5 ACE-5. Mortality and Clofibrate: Selection Bias?

A randomized placebo-controlled trial was carried out to estimate the effect of Clofibrate (a cholesterol-lowering drug) on 5-year mortality in men aged 30 to 64 years who had a recent myocardial infarct. The mortality in the Clofibrate group was 18.2 % and that in the control group was 19.4 %. This difference was not statistically significant. Subsequent analysis showed that about one-third of men given Clofibrate did not take their medication as directed. If these men are omitted from the analysis, there is a statistically significant effect of Clofibrate, with mortality in the Clofibrate group being 15.0 %.

• a. Which of the above two alternative analyses is more appropriate? Why?

• b. Is this a selection bias issue? Explain.

1.6 ACE-6. Identifying Study Subjects

A study entitled “Antidepressant Medication and Breast Cancer Risk” (Amer. J. of Epi, late 1990’s) stated in the methods section of the paper that “Cases were an age-stratified (< 50 and ≥ 50 years of age) random sample of women aged 25-74 years diagnosed with primary breast cancer during 1995 and 1996 (pathology report confirmed) and recorded in the population-based Ontario Cancer Registry. As the 1-year survival for breast cancer is 90 %, surrogate respondents were not used. Population controls, aged 25-74 years, were randomly sampled from the property assessment rolls of the Ontario Ministry of Finance; this database includes all home owners and tenants and lists age, sex, and address.”

• a. Discuss the authors’ approach to the identification of cases with respect to the potential for selection bias.

• b. Discuss the authors’ approach to the identification of controls with respect to the potential for selection bias.

• c. Does the decision not to allow surrogate respondents potentially affect selection bias?

1.7 ACE-7. Selection Bias in Observational Studies

The following questions concern the assessment of selection bias in observational studies where the putative association between salt intake and hypertension is being investigated. It is hypothesized that persons who have a greater intake of salt are at greater risk of developing chronic hypertension. It is assumed that both variables are dichotomized: high and low salt intake; hypertensive (DBP ≥ 95 and SPB ≥ 160) and normotensive. In considering the following questions, ignore the possibility of either information bias or confounding. Also, assume that the measure of effect in the source population is greater than one.

• a. Consider a case-control study in which equal numbers of hypertensives (cases) and normotensives (controls) are enrolled. Suppose that hypertensives from the study are gathered from an outpatient clinic designed to screen for hypertensives in the population, using the clinic on a volunteer basis. Normotensives are selected randomly from the surrounding population served by the clinic. Suppose that high-salt-intake hypertensives are four times as likely to be screened as low-salt-intake hypertensives in the clinic and that the sample of normotensives from the surrounding community is indeed a representative sample. What can be said about selection bias in estimating the exposure odds ratio (EOR)? (Formulate your answer in terms of the selection probabilities α, β, γ, and δ, or the selection odds r_D and r_{not D}.)

• b. In a type of case-control study similar to that described in part a, suppose that high- and low-salt-intake hypertensives are equally likely to be screened but that high-salt-intake normotensives in the community are more likely to cooperate in the study than are low-salt-intake normotensives in the community. What can be said about selection bias in estimating the EOR? (Formulate your answer in terms of the selection probabilities α, β, γ, and δ, or the selection odds r_D and r_{not D}.)

Answers to Study Questions and Quizzes

2.1 Q8.1

• 1. 2

• 2. 1; exposed cases are likely to be over-represented in the study when compared to unexposed cases.

2.2 Q8.2

• 1. Withdrawals from the study can distort the final sample to be analyzed as compared to the random sample obtained at the start of the trial.

• 2. Those who withdraw from the study have an unknown outcome and therefore cannot be analyzed.

• 3. A general population of patients with the specified cancer and eligible to receive either the standard or the new treatments.

• 4. The (expected) sample ignoring random error that would be obtained after the withdrawals from the random sample initially selected for the trial.

• 5. The source population is the population cohort from which the cases would be derived if an appropriate cohort study had been carried out.

• 6. The study population is the expected sample obtained from the cross-sectional sample that is retained for analysis. Alternatively, the study population is the stable population from which the cross-sectional sample is obtained for study.

2.3 Q8.3

• 1. It is unlikely that women in this control group (e.g., with cervical or ovarian cancers) would be selectively screened for their cancer from vaginal bleeding caused by estrogen use.

• 2. Away from the null because of selective screening of cases but not controls. This would yield too high a proportion of estrogen users among cases but a correct estimate of the proportion of estrogen users among controls.

• 3. Because those who have vaginal bleeding from using estrogen will be more likely to have their benign endometrial tumor detected than those non-users with benign endometrial tumors.

• 4. Using benign endometrial tumors as the control group would hopefully compensate for the selective screening of cases. However, it is not clear that the extent of selective screening would be the “same” for both cases and controls.

• 5. Having a benign tumor in the endometrium is not readily detected without vaginal bleeding. Therefore, controls with benign endometrial tumors who use estrogen are more likely to have their tumor detected than would nonuser controls.

• 6. Towards the null because there would be selective screening of controls but not cases. This would yield too high a proportion of estrogen users among controls but a correct estimate of the proportion of estrogen users among cases.

• 7. Because there is unlikely to be selective screening in the detection of control cases (with other gynecological cancers) when comparing estrogen users to nonusers.

2.4 Q8.4

• 1. No, since the risk ratio for the expected sample is 3, which equals the risk ratio in the source population.

• 2. Study population, because the sample just described gives the expected number of subjects obtained in the final sample.

• 3. The source is the population from which the initial 10 % sample obtained prior to follow-up was selected. This is the population of subjects from which cases were derived.

• 4. There is no selection bias because the RR = 3 in the study population, the same as in the source population.

• 5. Yes, because the estimated risk ratio in the study population of 3.9 is somewhat higher than (3) in the source population as a result of subjects being lost to follow-up.

• 6. No, the risk for exposed persons is 150/10,000 or.0150 in the source population and is 14/800 = .0175 in the sample.

• 7. No, the risk for unexposed persons is 50/10,000 or.0050 in the source population and 4/900 or.0044 in the sample.

• 8. No. There will only be selection bias if loss to follow-up results in risks for disease in the exposed and/or unexposed groups that are different in the final sample than in the original cohort.

2.5 Q8.5

• 1. Workers tend to be healthier than those in the general population and may therefore have a more favorable outcome regardless of exposure status.

• 2. Volunteers may have different characteristics from person who do not volunteer. The study population here is restricted to volunteers, whereas the source population is population-based, e.g., a community.

• 3. There is lack of external validity in drawing conclusions from a source population of volunteers to an external population that is population-based.

• 4. Yes. Clinic-based studies may lead to spurious conclusions because patients from clinics tend to have more severe illness than persons in a population-based sample.

2.6 Q8.6

• 1. α = 12/150 = .08; β = 4.5/50 = .09; γ = 788/9850 = .08; and δ = 895.5/9950 = .09.

• 2. No. The odds ratio in both source and study populations is 4.03. The risk ratio in both target and study populations is 3.0.

• 3. (α x δ) / (β x γ) = (.08 x.09) / (.09 x.08) = 1. This result illustrates a rule (described in the next activity) that says that the cross-product ratio of selection probabilities equal 1 if there is no bias in the odds ratio.

• 4. α = 14/150 = .093; β = 4/50 = .08; γ = 786/9850 = .08; and δ = 896/9950 = .09.

• 5. Yes. The odds ratio in the source population is 3.03 and in the study population is 4.00. The risk ratio in the source population is 3.0 and in the study population 3.9.

• 6. (α x δ) / (β x γ) = (.093 x.09) / (.08 x.08) = 1.3. This result illustrates a rule (described in the next activity) that says that the cross-product ratio of selection probabilities will differ from 1 if there is bias in the odds ratio.

2.7 Q8.7

• 1. No. The cross product ratio of selection ratios equals 1.

• 2. Yes. The cross product ratio of selection ratios equals (.093 x.09)/(.08 x.08) = 1.3, which is larger than 1.

• 3. Since the bias is defined as OR^o – OR, this means that the bias is away from the null (i.e., OR^o is further away from the null than is OR).

2.8 Q8.8

• 1. α = 100/100 = 1.0; β = 50/50 = 1.0; γ = 7920/9900 = .8; and δ = 7960/9950 = .8

• 2. Yes, the study population is a subset of the source population.

• 3. Yes, more or less; the incidence of disease in the overall source population is 150/20,000 = .0075 and 150/16,030 = .0093 in the overall study population. Within each exposure group, the risks are 100/10,000 = .01 and 50/110,000 = .005, respectively, in the source population; and 100/8,020 = .0124 and 50/8,010 = .0062, respectively, in the study population.

• 4. No. The cross product ratio of selection ratios equal 1.

• 5. No. The risk ratio in the study population is 1.998, which is not identical but essentially equal to the risk ratio in the source population.

• 6. α = 100/100 = 1.0; β = 50/50 = 1.0; γ = 99/9,900 = .01; and δ = 99.5/9950 = .01.

• 7. No. The cross-product ratio of selection ratios equals 1.

• 8. Yes. The risk ratio in the study population is 1.503, which is smaller than the risk ratio of 2.0 in the source population. The bias is towards the null.

• 9. No. The disease incidence is not at all rare in the study population. Overall the disease incidence is 150/348.5 = .4304 in the study population, and is 100/199 = .5025 and 50/149.5 = .3344 within exposed and unexposed groups, respectively.

• 10. α = 40/40 = 1.0; β = 10/10 = 1.0; γ = 30/60 = .5; and δ = 90/90 = 1.0.

• 11. Yes. The cross product ratio of selection ratios equals (1 x 1)/(1 x.5) = 2, which is clearly different from 1. The incorrect OR = 12, which is greater than the correct OR = 6, indicating that the bias is away from the null.

• 12. Yes. The risk ratio in the study population is 5.71, which is larger than the risk ratio of 4.0 in the source population. The bias is away from the null.

• 13. The disease is not rare in either source or study populations. It is particularly not rare for exposed subjects in the study population.

• 14. No. The bias in the odds ratio (OR^o =12 whereas OR = 6) is much larger than the bias in the risk ratio (RR^o =5.71 whereas RR =4.0).

2.9 Q8.9

• 1. α > β

• 2. γ > δ

• 3. α = β

• 4. γ = δ

2.10 Q8.10

• 1. True. Unexposed controls in the study will include persons with sexual partners with and without STD’s. Unexposed non-cases in the source population will not include sexual partners without STDs.

• 2. False. Exposed controls will go to STD clinics primarily for reasons of disease prevention rather than contraception. They are more likely to have sex partners with STDs than unexposed controls, and therefore, are more likely to represent the source population.

• 3. Yes. In the source population the odds for condom users in one-half the odds for non-users, whereas the odds ratio is one in the study population.

• 4. Yes. The ratio (i.e., odds) of unexposed to exposed non-cases in the study population is 6 to 1 whereas the odds of unexposed to exposed non-cases in the source population is 3 to 1.

• 5. α = 100/100 = 1; β = 600/600 = 1; γ = 200/200 = 1; and δ = 1200/600 = 2.

• 6. In this example, α (1) equals β (1), but γ (1) is less than δ (2). Consequently, the cross product ratio of selection ratios should be greater than 1. Thus, there is selection bias in the odds ratio.

• 7. The bias would be towards the null, since the (incorrect) OR in the study population is equal to 1 whereas the correct OR is different (in this case, smaller) than 1 in the source population.

• 8. False. Both unexposed controls in the study and unexposed non-cases will be restricted to have sexual partners with STD’s.

• 9. False. Exposed controls and exposed non-cases will also be restricted to have sexual partners with STD’s. Consequently, exposed controls in the study will reflect the source population of exposed non-cases, all those partners have STD’s.

• 10. γ, the selection ratio for exposed controls, is equal to δ, the selection ratio of unexposed controls.

• 11. The cross product of selection ratios should be equal to 1.

2.11 Q8.11

• 1. The source population is the community population and the study population is the expected case-control sample under the selection conditions described for the study.

• 2. α = 225/400 = .5625; β = 75/200 = .375; γ = 100/100,000 = .001; and δ = 200/200,000 = .001.

• 3. Yes, because each is derived as a ratio in which the numerator is a subset of the denominator.

• 4. α = .5625 is greater than β = .375, whereas γ = δ. Consequently, the cross-product ratio of selection probabilities = (.5625 x.001)/(.375 x.001) = 1.5, which is different from 1.

• 5. The OR in the study population is 6 whereas the OR in the source population is 4. The incorrect (i.e., biased) odds ratio of 6 is an overestimate of the correct odds ratio, and is therefore away from the null.

• 6. Patients who are both cases and smokers are over-represented in the study when compared to the community population.

• 7. Yes. Controls are split 50:50 among exposed and unexposed, whereas non-cases in the community are split 1:2.

• 8. α = 225/400 = .5625; β = 75/200 = .375; γ = 150/100,000 = .0015; and δ = 150/200,000 = .00075.

• 9. α = .5625 is greater than β = .375, and γ = .0015 is less than δ = .00075. The cross-product ratio of selection probabilities = (.5625 x.00075)/(.375 x.0015) = 0.75, which is different from 1.

• 10. The OR in the study population is 3 whereas the OR in the source population is 4. The incorrect (i.e., biased) odds ratio of 3 is an underestimate of the correct odds ratio, and is therefore towards the null.

• 11. Hospital patients who are both cases and smokers are more over-represented than are hospital patients who are both controls and smokers when compared to the community population.

2.12 Q8.12

• 1. 25/50 = .5 for exposed and 40/80 = .5 for unexposed.

• 2. 142.5/150 = .95

• 3. 540/720 = .75

• 4. α = β = .5; γ = .95, and δ = .75.

• 5. (.5 x.75)/(.95 x 5) = .79

• 6. Yes, a slight bias. For the odds ratio, ROR = 3 whereas POR (= OR in the study population) = 2.4. For the risk ratio, RR 2.5 whereas PR (= effect in the study population) = 2.2. The bias is towards the null.

2.13 Q8.13

• 1. prior to

• 2. non-response, study population

• 3. one

• 4. earlier, by

• 5. prevalence data, incidence data

• 6. selective survival

• 7. survivors

• 8. survive

2.14 Q8.14

• 1. ${\rm \hat A} = {\rm 40, \hat B } = {\rm 10, \hat C } = {\rm 60, \hat D } = {\rm 90}$

• 2. OR_adj = 6

• 3. ${\rm \hat r}_{\rm D} {\rm } = {\rm 1, \hat r}_{{\rm \bar D}} =.5$

• 4. $\begin{array}{l} {\rm OR}_{{\rm adj}} = {\rm O\hat R } \times {\rm (\hat r}_{{\rm \bar D}} /{\rm \hat r}_{\rm D} ){\rm } = {\rm 12 } \times \\ \\ {\rm (}{\rm.5/1) } = {\rm 6} \\ \end{array}$ RR_adj = (40/100)/(10/100) = 4

• 5. ${\rm \hat r}_{\rm E} {\rm } = {\rm \hat \alpha /\hat \gamma } = {\rm 2, \hat r}_{{\rm \bar E}} = {\rm \hat \beta /\hat \delta } = {\rm 1}$

• 6. RR_adj = [40/(40 + [30 x 2])]/{10/(10 + [90 x 1])} = 4

2.15 Q8.15

• 1. A decision will have to be made as to which of the two control groups is the most suitable, e.g., which control group is more representative of the source population.

• 2. Either of the following is possible: 1.) There is no selection bias because both estimated effects are correct; or 2.) There is selection bias because both estimated effects are biased.

• 3. Yes. Since the presence of selection bias depends on the selection parameters within the 2x2 table, selection bias may still occur even with excellent response rates and minimal loss-to-follow-up.

• 4. Not necessarily. As with the previous question, the presence of selection bias depends on the selection parameters within the 2x2 table, whereas the information provided only considers follow-up loss on the marginals (i.e., total exposed and total unexposed) of the 2x2 table.

• 5. No. The magnitude of selection bias cannot be determined without guestimates of the selection parameters or their ratios.

• 6. No. A corrected odds ratio would require guestimates of the selection parameters or their ratios.

• 7. Since α is less than β and gamma equals delta, the cross-product (α x δ) / (β x γ) must be less than one. Thus, OR^o must be less than OR, so that the bias would be towards the null if the OR is greater than 1 and away from the null if the OR is less than 1.

• 8. No. Without knowing whether α over γ is either equal to, less than, or greater than β over δ, it is not possible to determine whether the cross-product (α x δ)/(β x γ) = (α / γ)/(β / δ) is equal to, less than, or greater than 1.

• 9. Yes. There is no selection bias in the odds ratio because the cross-product (α x δ)/(β x γ) equals 1.

• 10. Yes. The bias must be away from the null because the cross-product (α x δ)/(β x γ) is greater than 1 so that the biased OR^o is greater than the correct OR, which is greater than 1.

2.16 Q8.16

• 1. adjusted cell frequencies, selection parameter

• 2. estimated odds ratio, estimated cross-product of selection parameters

• 3. incident cases, prevalent cases

• 4. nested case-control, hospital-based

• 5. high response, loss-to-follow-up

• 6. True

• 7. True – Since the “worst-case” analysis can demonstrate that the worst amount of bias will have a negligible effect on the conclusions, one could rule out selection bias. However, since the “worst-case” analysis gives us the “worst possible” results, we cannot confirm selection bias. We cannot be sure that our results will be as extreme as “worst-case” results.

• 8. 32, 22, 110, 106

• 9. 3.9

• 10. 1.3

• 11. Yes

• 12. No - A worst-case analysis gives the “worst possible” results. Therefore, we cannot be sure that the lost-to-follow-up results that “actually” occur are as extreme as the worst-case “possible”.