Skip to main content
Log in

Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression

  • Published:
Psychometrika Aims and scope Submit manuscript

An Erratum to this article was published on 05 June 2013

Abstract

The study of prediction bias is important and the last five decades include research studies that examined whether test scores differentially predict academic or employment performance. Previous studies used ordinary least squares (OLS) to assess whether groups differ in intercepts and slopes. This study shows that OLS yields inaccurate inferences for prediction bias hypotheses. This paper builds upon the criterion-predictor factor model by demonstrating the effect of selection, measurement error, and measurement bias on prediction bias studies that use OLS. The range restricted, criterion-predictor factor model is used to compute Type I error and power rates associated with using regression to assess prediction bias hypotheses. In short, OLS is not capable of testing hypotheses about group differences in latent intercepts and slopes. Additionally, a theorem is presented which shows that researchers should not employ hierarchical regression to assess intercept differences with selected samples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1.
Figure 2.
Figure 3.
Figure 4.

Similar content being viewed by others

References

  • AERA, APA, NCME (1999). Standards for educational and psychological testing. Washington: American Educational Research Association.

    Google Scholar 

  • Aguinis, H., Boik, R.J., & Pierce, C.A. (2001). A generalized solution for approximating the power to detect effects of categorical moderator variables using multiple regression. Organizational Research Methods, 4, 291–323.

    Article  Google Scholar 

  • Aguinis, H., Boik, R.J., Pierce, C.A., & Beaty, J.C. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. Journal of Applied Psychology, 90, 94–107.

    Article  PubMed  Google Scholar 

  • Aguinis, H., Culpepper, S.A., & Pierce, C.A. (2010). Revival of test bias research in preemployment testing. Journal of Applied Psychology, 95, 648–680.

    Article  PubMed  Google Scholar 

  • Aguinis, H., & Stone-Romero, E.F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192–206.

    Article  Google Scholar 

  • Aitkin, M. (1964). Correlation in a singly truncated bivariate normal distribution. Psychometrika, 29, 263–270.

    Article  Google Scholar 

  • Arnold, B.C., Beaver, R.J., Groeneveld, R.A., & Meeker, W.Q. (1993). The nontruncated marginal of a truncated bivariate normal distribution. Psychometrika, 58, 471–488.

    Article  Google Scholar 

  • Balakkrishnan, N., & Lai, C. (2009). Continuous bivariate distributions. New York: Springer.

    Google Scholar 

  • Barr, D.R., & Sherrill, E.T. (1999). Mean and variance of truncated normal distributions. The American Statistician, 53, 357–361.

    Google Scholar 

  • Bartlett, C.J., Bobko, P., Mosier, S.B., & Hannan, R. (1978). Testing for fairness with a moderated regression strategy: an alternative to differential analysis. Personnel Psychology, 31, 233–241.

    Article  Google Scholar 

  • Birnbaum, Z.W., Paulson, E., & Andrews, F.C. (1950). On the effect of selection performed on some coordinates of a multi-dimensional population. Psychometrika, 15, 191–204.

    Article  PubMed  Google Scholar 

  • Borsboom, D., Romeijn, J., & Wicherts, J.M. (2008). Measurement invariance versus selection invariance: is fair selection possible? Psychological Methods, 13, 75–98.

    Article  PubMed  Google Scholar 

  • Cleary, T.A. (1968). Test bias: prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115–124.

    Article  Google Scholar 

  • Culpepper, S.A. (2010). Studying individual differences in predictability with gamma regression and nonlinear multilevel models. Multivariate Behavioral Research, 45, 153–185.

    Article  Google Scholar 

  • Culpepper, S.A. (in press). Evaluating EIV, OLS, and SEM estimators of group slope differences in the presence of measurement error: the single indicator case. Applied Psychological Measurement.

  • Culpepper, S.A., & Aguinis, H. (2011). Analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166–178.

    Article  PubMed  Google Scholar 

  • Culpepper, S.A., & Davenport, E.C. (2009). Assessing differential prediction of college grades by race/ethnicity with a multilevel model. Journal of Educational Measurement, 46, 220–242.

    Article  Google Scholar 

  • DeCorte, W., Lievens, F., & Sackett, P.R. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. Journal of Applied Psychology, 92, 1380–1393.

    Article  Google Scholar 

  • Dorans, N.J. (2004). Using subpopulation invariance to assess test score equity. Journal of Educational Measurement, 41, 43–68.

    Article  Google Scholar 

  • Finch, D.M., Edwards, B.D., & Wallace, J.C. (2009). Multistage selection strategies: simulating the effects on adverse impact and expected performance for various predictor combinations. Journal of Applied Psychology, 94, 318–340.

    Article  PubMed  Google Scholar 

  • Gratz v. Bollinger (2003). 539 U.S. 244.

  • Grutter v. Bollinger (2003). 539 U.S. 306.

  • Holmes, D.J. (1990). The robustness of the usual correction for restriction in range due to explicit selection. Psychometrika, 55, 19–32.

    Article  Google Scholar 

  • Humphreys, L. (1952). Individual differences. Annual Review of Psychology, 3, 131–150.

    Article  PubMed  Google Scholar 

  • Humpreys, L.G. (1986). An analysis and evaluation of test and item bias in the prediction context. Journal of Applied Psychology, 71, 327–333.

    Article  Google Scholar 

  • Kuncel, N.R., Hezlett, S.A., & Ones, D.S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Psychological Bulletin, 127, 162–181.

    Article  PubMed  Google Scholar 

  • Lautenschlager, G.J., & Mendoza, J.L. (1986). A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction. Applied Psychological Measurement, 10, 133–139.

    Article  Google Scholar 

  • Lawley, D.N. (1943). A note on Karl Pearson’s selection formulae. Proceedings of the Royal Society of Edinburgh. Section A, Mathematical and Physical Sciences 62, 28–30.

    Google Scholar 

  • Linn, R.L. (1983). Pearson selection formulas: implications for studies of predictive bias and estimates of educational effects in selected samples. Journal of Educational Measurement, 20, 1–15.

    Article  Google Scholar 

  • Linn, R.L. (1984). Selection bias: multiple meanings. Journal of Educational Measurement, 21, 33–47.

    Article  Google Scholar 

  • Linn, R.L., & Werts, C.E. (1971). Considerations for studies of test bias. Journal of Educational Measurement, 8, 1–4.

    Article  Google Scholar 

  • Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525–543.

    Article  Google Scholar 

  • Millsap, R.E. (1997). Invariance in measurement and prediction: their relationship in the single-factor case. Psychological Methods, 2, 248–260.

    Article  Google Scholar 

  • Millsap, R.E. (1998). Group differences in regression intercepts: implications for factorial invariance. Multivariate Behavioral Research, 33, 403–424.

    Article  Google Scholar 

  • Millsap, R.E. (2007). Invariance in measurement and prediction revisited. Psychometrika, 72, 461–473.

    Article  Google Scholar 

  • Millsap, R.E., & Kwok, O.M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93–115.

    Article  PubMed  Google Scholar 

  • Muthen, B. (1990). Moments of the censored and truncated bivariate normal distribution. British Journal of Mathematical & Statistical Psychology, 43, 131–143.

    Article  Google Scholar 

  • Pearson, K. (1903). Mathematical contributions to the theory of evolution—xi. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions, CC.–A 321, 1–66.

    Google Scholar 

  • Ricci v. DeStefano (2009). 557 U.S. 244.

  • Rosenbaum, S. (1961). Moments of a truncated bivariate normal distribution. Journal of the Royal Statistical Society. Series B. Methodological, 23, 405–408.

    Google Scholar 

  • Saad, S., & Sackett, P.R. (2002). Investigating differential prediction by gender in employment-oriented personality measures. Journal of Applied Psychology, 87, 667–674.

    Article  PubMed  Google Scholar 

  • SIOP (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green: Society for Industrial and Organizational Psychology, Inc.

    Google Scholar 

  • Wicherts, J.M., & Millsap, R.E. (2009). The absence of underprediction does not imply the absence of measurement bias. The American Psychologist, 64, 281–283.

    Article  PubMed  Google Scholar 

  • Young, J.W. (2001). Differential validity, differential prediction, and college admission testing: a comprehensive review and analysis (Research Report No. 6). College Entrance Examination Board.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Steven Andrew Culpepper.

Additional information

An erratum to this article is available at http://dx.doi.org/10.1007/s11336-013-9345-1.

Appendix: Additional Derivations

Appendix: Additional Derivations

1.1 A.1 Observed Prediction Invariance Under the Criterion-Predictor Factor Model

This subsection derives the conditions in (13) and (14) for observed prediction invariance under the criterion-predictor factor model. The condition for slope invariance is fairly straightforward to derive. Specifically, the observed OLS slope for group i is the ratio of the covariance between Z i and Y i to the variance of Z i from (12),

$$ \beta_{1i}= \frac{\lambda_{yi}\varGamma_i \phi_i \lambda_{zi}}{\lambda_{zi}^2 \phi_i +\varTheta_{zi}}. $$
(A.1)

Consequently, the condition for slope invariance in (14) arises when β 1i =B1.

Subgroup intercepts are defined as β 0i =E(Y i )−β 1i E(Z i ). Substituting the definitions of E(Y i ), E(Z i ), and β 1i into the definition of β 0i and rearranging terms yields,

(A.2)

Subgroup intercept invariance occurs when β 0i =B0, which confirms (13).

1.2 A.2 Derivations to Support Theorem 1

An expression for β 1s (HR) is defined in the first element of \(\varSigma_{zGs}^{-1}\varSigma_{zGys}\) where Σ zGs is the variance–covariance matrix of Z s and G and Σ zGys is a vector of covariances between Z s and Y s and G and Y s . Multiplying the elements in \(\varSigma_{zGs}^{-1}\varSigma_{zGys}\) implies that β 1s (HR) is defined as:

(A.3)

Results concerning finite mixtures imply that σ zys and \(\sigma_{zs}^{2}\) can be written as functions of p s and subgroup covariances and variances. Specifically, σ zys and \(\sigma_{zs}^{2}\) are defined as

(A.4)
(A.5)

Consequently, β 1s (HR) is defined as

$$ \beta_{1s} (HR )= \frac{p_s\sigma_{zy1s}+ (1-p_s )\sigma_{zy2s}}{p_s\sigma_{z1s}^2+ (1-p_s )\sigma_{z2s}^2}. $$
(A.6)

Observed subgroup slopes will equal the common slope only in the absence of observed slope differences. For instance, β 11s =β 1s (HR) if and only if β 11s =β 12s . More formally, β 11s =β 1s (HR) when,

(A.7)

Cross multiplying in (A.7) and simplifying terms implies that β 11s =β 1s (HR) only when β 11s =β 12s . Additionally, β 11s >β 1s (HR) if β 11s >β 12s and vice versa.

1.3 A.3 Derivation of Correlations Among Variables

This subsection includes additional derivations to support several equations used to compute \(\sigma_{W_{y}}^{2}\) and to compute the probability of detecting intercept and slope differences with OLS. Note that the derivations employ the use of finite mixtures and the fact that G is a categorical variable that codes group membership using zero or one. The derivations of correlations proceed by specifying covariances between variables as the expected value of the product of the random variables less the product of their expected values. The first subsection presents derivations for computing correlations necessary to compute \(\sigma_{W_{y}}^{2}\) and the second subsection includes derivations to compute Type I error and power rates of OLS tests of intercept and slope differences.

1.3.1 A.3.1 Derivations of Correlations for Computing \(\sigma_{W_{y}}^{2}\)

This section derives expressions for \(\sigma_{W_{z}G}^{2}\), \(\rho_{W_{z},W_{z}G}\), and \(\rho_{G,W_{z}G}\). First, the variance of the product W z G is,

(A.8)

Substituting the definitions of ϕ 1 and κ 1 from (5) and (7) into the above equation yields

(A.9)

By definition, the E(W z )=0 and \(E (W_{z}^{2} )=1\) across the two groups, so the correlation between W z and W z G is,

(A.10)

The correlation between G and W z G can be found in a similar manner. Specifically, \(\rho_{G,W_{z}G}\) is,

(A.11)

1.3.2 A.3.2 Derivations of Correlations Among Z s , G, Z s G, and Y s

This section derives expressions for ρ z,zGs , ρ G,zGs , and ρ zG,ys . Note that expressions for ρ z,Gs and ρ y,Gs are not derived, because these correlations are simply biserial correlations. It is important to first derive \(\sigma_{zGs}^{2}\), which is equal to

(A.12)

where the last equality was obtained by substituting μ z1s =μ zs +(1−p s μ zs into (A.12) and factoring out a 1−p s from the similar terms.

The correlation between Z s and Z s G is,

(A.13)

Substituting the definition of μ zs1 into (A.13) and rearranging terms produces the result in (41). The correlation between G and Z s G is similarly expressed as the covariance divided by respective standard deviations:

(A.14)

where the last equation was established by substituting for μ z1s and factoring out 1−p s .

Rights and permissions

Reprints and permissions

About this article

Cite this article

Culpepper, S.A. Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression. Psychometrika 77, 561–580 (2012). https://doi.org/10.1007/s11336-012-9270-8

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-012-9270-8

Key words

Navigation