Skip to main content

Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression

An Erratum to this article was published on 05 June 2013

Abstract

The study of prediction bias is important and the last five decades include research studies that examined whether test scores differentially predict academic or employment performance. Previous studies used ordinary least squares (OLS) to assess whether groups differ in intercepts and slopes. This study shows that OLS yields inaccurate inferences for prediction bias hypotheses. This paper builds upon the criterion-predictor factor model by demonstrating the effect of selection, measurement error, and measurement bias on prediction bias studies that use OLS. The range restricted, criterion-predictor factor model is used to compute Type I error and power rates associated with using regression to assess prediction bias hypotheses. In short, OLS is not capable of testing hypotheses about group differences in latent intercepts and slopes. Additionally, a theorem is presented which shows that researchers should not employ hierarchical regression to assess intercept differences with selected samples.

This is a preview of subscription content, access via your institution.

Figure 1.
Figure 2.
Figure 3.
Figure 4.

References

  1. AERA, APA, NCME (1999). Standards for educational and psychological testing. Washington: American Educational Research Association.

    Google Scholar 

  2. Aguinis, H., Boik, R.J., & Pierce, C.A. (2001). A generalized solution for approximating the power to detect effects of categorical moderator variables using multiple regression. Organizational Research Methods, 4, 291–323.

    Article  Google Scholar 

  3. Aguinis, H., Boik, R.J., Pierce, C.A., & Beaty, J.C. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. Journal of Applied Psychology, 90, 94–107.

    PubMed  Article  Google Scholar 

  4. Aguinis, H., Culpepper, S.A., & Pierce, C.A. (2010). Revival of test bias research in preemployment testing. Journal of Applied Psychology, 95, 648–680.

    PubMed  Article  Google Scholar 

  5. Aguinis, H., & Stone-Romero, E.F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192–206.

    Article  Google Scholar 

  6. Aitkin, M. (1964). Correlation in a singly truncated bivariate normal distribution. Psychometrika, 29, 263–270.

    Article  Google Scholar 

  7. Arnold, B.C., Beaver, R.J., Groeneveld, R.A., & Meeker, W.Q. (1993). The nontruncated marginal of a truncated bivariate normal distribution. Psychometrika, 58, 471–488.

    Article  Google Scholar 

  8. Balakkrishnan, N., & Lai, C. (2009). Continuous bivariate distributions. New York: Springer.

    Google Scholar 

  9. Barr, D.R., & Sherrill, E.T. (1999). Mean and variance of truncated normal distributions. The American Statistician, 53, 357–361.

    Google Scholar 

  10. Bartlett, C.J., Bobko, P., Mosier, S.B., & Hannan, R. (1978). Testing for fairness with a moderated regression strategy: an alternative to differential analysis. Personnel Psychology, 31, 233–241.

    Article  Google Scholar 

  11. Birnbaum, Z.W., Paulson, E., & Andrews, F.C. (1950). On the effect of selection performed on some coordinates of a multi-dimensional population. Psychometrika, 15, 191–204.

    PubMed  Article  Google Scholar 

  12. Borsboom, D., Romeijn, J., & Wicherts, J.M. (2008). Measurement invariance versus selection invariance: is fair selection possible? Psychological Methods, 13, 75–98.

    PubMed  Article  Google Scholar 

  13. Cleary, T.A. (1968). Test bias: prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115–124.

    Article  Google Scholar 

  14. Culpepper, S.A. (2010). Studying individual differences in predictability with gamma regression and nonlinear multilevel models. Multivariate Behavioral Research, 45, 153–185.

    Article  Google Scholar 

  15. Culpepper, S.A. (in press). Evaluating EIV, OLS, and SEM estimators of group slope differences in the presence of measurement error: the single indicator case. Applied Psychological Measurement.

  16. Culpepper, S.A., & Aguinis, H. (2011). Analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166–178.

    PubMed  Article  Google Scholar 

  17. Culpepper, S.A., & Davenport, E.C. (2009). Assessing differential prediction of college grades by race/ethnicity with a multilevel model. Journal of Educational Measurement, 46, 220–242.

    Article  Google Scholar 

  18. DeCorte, W., Lievens, F., & Sackett, P.R. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. Journal of Applied Psychology, 92, 1380–1393.

    Article  Google Scholar 

  19. Dorans, N.J. (2004). Using subpopulation invariance to assess test score equity. Journal of Educational Measurement, 41, 43–68.

    Article  Google Scholar 

  20. Finch, D.M., Edwards, B.D., & Wallace, J.C. (2009). Multistage selection strategies: simulating the effects on adverse impact and expected performance for various predictor combinations. Journal of Applied Psychology, 94, 318–340.

    PubMed  Article  Google Scholar 

  21. Gratz v. Bollinger (2003). 539 U.S. 244.

  22. Grutter v. Bollinger (2003). 539 U.S. 306.

  23. Holmes, D.J. (1990). The robustness of the usual correction for restriction in range due to explicit selection. Psychometrika, 55, 19–32.

    Article  Google Scholar 

  24. Humphreys, L. (1952). Individual differences. Annual Review of Psychology, 3, 131–150.

    PubMed  Article  Google Scholar 

  25. Humpreys, L.G. (1986). An analysis and evaluation of test and item bias in the prediction context. Journal of Applied Psychology, 71, 327–333.

    Article  Google Scholar 

  26. Kuncel, N.R., Hezlett, S.A., & Ones, D.S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Psychological Bulletin, 127, 162–181.

    PubMed  Article  Google Scholar 

  27. Lautenschlager, G.J., & Mendoza, J.L. (1986). A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction. Applied Psychological Measurement, 10, 133–139.

    Article  Google Scholar 

  28. Lawley, D.N. (1943). A note on Karl Pearson’s selection formulae. Proceedings of the Royal Society of Edinburgh. Section A, Mathematical and Physical Sciences 62, 28–30.

    Google Scholar 

  29. Linn, R.L. (1983). Pearson selection formulas: implications for studies of predictive bias and estimates of educational effects in selected samples. Journal of Educational Measurement, 20, 1–15.

    Article  Google Scholar 

  30. Linn, R.L. (1984). Selection bias: multiple meanings. Journal of Educational Measurement, 21, 33–47.

    Article  Google Scholar 

  31. Linn, R.L., & Werts, C.E. (1971). Considerations for studies of test bias. Journal of Educational Measurement, 8, 1–4.

    Article  Google Scholar 

  32. Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525–543.

    Article  Google Scholar 

  33. Millsap, R.E. (1997). Invariance in measurement and prediction: their relationship in the single-factor case. Psychological Methods, 2, 248–260.

    Article  Google Scholar 

  34. Millsap, R.E. (1998). Group differences in regression intercepts: implications for factorial invariance. Multivariate Behavioral Research, 33, 403–424.

    Article  Google Scholar 

  35. Millsap, R.E. (2007). Invariance in measurement and prediction revisited. Psychometrika, 72, 461–473.

    Article  Google Scholar 

  36. Millsap, R.E., & Kwok, O.M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93–115.

    PubMed  Article  Google Scholar 

  37. Muthen, B. (1990). Moments of the censored and truncated bivariate normal distribution. British Journal of Mathematical & Statistical Psychology, 43, 131–143.

    Article  Google Scholar 

  38. Pearson, K. (1903). Mathematical contributions to the theory of evolution—xi. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions, CC.–A 321, 1–66.

    Google Scholar 

  39. Ricci v. DeStefano (2009). 557 U.S. 244.

  40. Rosenbaum, S. (1961). Moments of a truncated bivariate normal distribution. Journal of the Royal Statistical Society. Series B. Methodological, 23, 405–408.

    Google Scholar 

  41. Saad, S., & Sackett, P.R. (2002). Investigating differential prediction by gender in employment-oriented personality measures. Journal of Applied Psychology, 87, 667–674.

    PubMed  Article  Google Scholar 

  42. SIOP (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green: Society for Industrial and Organizational Psychology, Inc.

    Google Scholar 

  43. Wicherts, J.M., & Millsap, R.E. (2009). The absence of underprediction does not imply the absence of measurement bias. The American Psychologist, 64, 281–283.

    PubMed  Article  Google Scholar 

  44. Young, J.W. (2001). Differential validity, differential prediction, and college admission testing: a comprehensive review and analysis (Research Report No. 6). College Entrance Examination Board.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Steven Andrew Culpepper.

Additional information

An erratum to this article is available at http://dx.doi.org/10.1007/s11336-013-9345-1.

Appendix: Additional Derivations

Appendix: Additional Derivations

A.1 Observed Prediction Invariance Under the Criterion-Predictor Factor Model

This subsection derives the conditions in (13) and (14) for observed prediction invariance under the criterion-predictor factor model. The condition for slope invariance is fairly straightforward to derive. Specifically, the observed OLS slope for group i is the ratio of the covariance between Z i and Y i to the variance of Z i from (12),

$$ \beta_{1i}= \frac{\lambda_{yi}\varGamma_i \phi_i \lambda_{zi}}{\lambda_{zi}^2 \phi_i +\varTheta_{zi}}. $$
(A.1)

Consequently, the condition for slope invariance in (14) arises when β 1i =B1.

Subgroup intercepts are defined as β 0i =E(Y i )−β 1i E(Z i ). Substituting the definitions of E(Y i ), E(Z i ), and β 1i into the definition of β 0i and rearranging terms yields,

(A.2)

Subgroup intercept invariance occurs when β 0i =B0, which confirms (13).

A.2 Derivations to Support Theorem 1

An expression for β 1s (HR) is defined in the first element of \(\varSigma_{zGs}^{-1}\varSigma_{zGys}\) where Σ zGs is the variance–covariance matrix of Z s and G and Σ zGys is a vector of covariances between Z s and Y s and G and Y s . Multiplying the elements in \(\varSigma_{zGs}^{-1}\varSigma_{zGys}\) implies that β 1s (HR) is defined as:

(A.3)

Results concerning finite mixtures imply that σ zys and \(\sigma_{zs}^{2}\) can be written as functions of p s and subgroup covariances and variances. Specifically, σ zys and \(\sigma_{zs}^{2}\) are defined as

(A.4)
(A.5)

Consequently, β 1s (HR) is defined as

$$ \beta_{1s} (HR )= \frac{p_s\sigma_{zy1s}+ (1-p_s )\sigma_{zy2s}}{p_s\sigma_{z1s}^2+ (1-p_s )\sigma_{z2s}^2}. $$
(A.6)

Observed subgroup slopes will equal the common slope only in the absence of observed slope differences. For instance, β 11s =β 1s (HR) if and only if β 11s =β 12s . More formally, β 11s =β 1s (HR) when,

(A.7)

Cross multiplying in (A.7) and simplifying terms implies that β 11s =β 1s (HR) only when β 11s =β 12s . Additionally, β 11s >β 1s (HR) if β 11s >β 12s and vice versa.

A.3 Derivation of Correlations Among Variables

This subsection includes additional derivations to support several equations used to compute \(\sigma_{W_{y}}^{2}\) and to compute the probability of detecting intercept and slope differences with OLS. Note that the derivations employ the use of finite mixtures and the fact that G is a categorical variable that codes group membership using zero or one. The derivations of correlations proceed by specifying covariances between variables as the expected value of the product of the random variables less the product of their expected values. The first subsection presents derivations for computing correlations necessary to compute \(\sigma_{W_{y}}^{2}\) and the second subsection includes derivations to compute Type I error and power rates of OLS tests of intercept and slope differences.

A.3.1 Derivations of Correlations for Computing \(\sigma_{W_{y}}^{2}\)

This section derives expressions for \(\sigma_{W_{z}G}^{2}\), \(\rho_{W_{z},W_{z}G}\), and \(\rho_{G,W_{z}G}\). First, the variance of the product W z G is,

(A.8)

Substituting the definitions of ϕ 1 and κ 1 from (5) and (7) into the above equation yields

(A.9)

By definition, the E(W z )=0 and \(E (W_{z}^{2} )=1\) across the two groups, so the correlation between W z and W z G is,

(A.10)

The correlation between G and W z G can be found in a similar manner. Specifically, \(\rho_{G,W_{z}G}\) is,

(A.11)

A.3.2 Derivations of Correlations Among Z s , G, Z s G, and Y s

This section derives expressions for ρ z,zGs , ρ G,zGs , and ρ zG,ys . Note that expressions for ρ z,Gs and ρ y,Gs are not derived, because these correlations are simply biserial correlations. It is important to first derive \(\sigma_{zGs}^{2}\), which is equal to

(A.12)

where the last equality was obtained by substituting μ z1s =μ zs +(1−p s μ zs into (A.12) and factoring out a 1−p s from the similar terms.

The correlation between Z s and Z s G is,

(A.13)

Substituting the definition of μ zs1 into (A.13) and rearranging terms produces the result in (41). The correlation between G and Z s G is similarly expressed as the covariance divided by respective standard deviations:

(A.14)

where the last equation was established by substituting for μ z1s and factoring out 1−p s .

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Culpepper, S.A. Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression. Psychometrika 77, 561–580 (2012). https://doi.org/10.1007/s11336-012-9270-8

Download citation

Key words

  • selection
  • prediction bias
  • measurement bias
  • type I error
  • power