Skip to main content
SpringerLink
Log in

Robust evaluation of fit indices to fake-good perturbation of ordinal data

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

This study extended the findings of a former simulation study (Multivar Behav Res 47:519–546, 2012) to evaluate the sensitivity of a large set of SEM-based fit indices to fake-good ordinal data. In the new simulation study we manipulated a comprehensive set of factors (including 3 robust estimation procedures and 3 different faking good models) that could influence the performance of 8 widely used fit indices. The simulation study conditions were chosen to highlight the differences among the fit indices, as well as to cover a wide variety of conditions. Our results demonstrated empirically that the normed fit index (NFI) turned out to be the most reliable fit index with a high sensitivity to fake perturbations. This result was evident in all the simulation design conditions except for those characterized by slight faking levels of perturbations. Interestingly, unlike NFI, the comparative fit index seemed to be highly insensitive to fake data when robust estimation conditions were considered. On the basis of the results of the simulation study we proposed a simple qualitative criterion to evaluate the impact of faking on statistical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Beauducel, A., Herzberg, P.Y.: On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Struct. Equ. Model. 13, 186–203 (2006)

    Article  Google Scholar 

  • Bentler, P.M.: Comparative fit indexes in structural models. Psychol. Bull. 107, 238–246 (1990)

    Article  Google Scholar 

  • Bentler, P.M.: EQS Structural Equations Program Manual. Multivariate Software, Encino (1995)

    Google Scholar 

  • Bentler, P.M., Bonett, D.G.: Significance tests and goodness of fit in the analysis of covariance structures. Psychol. Bull. 88, 588–606 (1980)

    Article  Google Scholar 

  • Browne, M.W., Cudeck, R.: Alternative ways of assessing model fit. In: Bollen, K.A., Long, J.S. (eds.) Testing Structural Equation Models, pp. 136–162. Sage, Beverly Hills (1993)

    Google Scholar 

  • Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Erlbaum Associates, Hillsdale (1988)

    Google Scholar 

  • Curran, P.J., Bollen, K.A., Paxton, P., Kirby, J., Chen, F.: The noncentral Chi-square distribution in misspecified structural equation models: Finite sample results from a Monte Carlo simulation. Multivar. Behav. Res. 37, 1–36 (2002)

    Article  Google Scholar 

  • Ding, L., Velicer, W.F., Harlow, L.L.: Effects of estimation methods, number of indicators per factor, and improper solutions on structural equation modeling fit indices. Struct. Equ. Model.: A Multidiscip. J. 2(2), 119–143 (1995)

    Article  Google Scholar 

  • Dobson, A.J.: An Introduction to Generalized Linear Models, 2nd edn. Chapman & Hall/CRC Press, Boca Raton (2002)

    Google Scholar 

  • Dolan, C.V.: Factor analysis of variables with 2, 3, 5 and 7 response categories: a comparison of categorical variable estimators using simulated data. Br. J. Math. Stat. Psychol. 47, 309–326 (1994)

    Article  Google Scholar 

  • Donovan, J.J., Dwight, S.A., Schneider, D.: The impact of applicant faking on selection measures, hiring decisions, and employee performance. J. Bus. Psychol. 29, 1–15 (2014)

    Article  Google Scholar 

  • Fan, X., Felsovalyi, A., Sivo, S.A., Keenan, S.: SAS for Monte Carlo Studies: a Guide for Quantitative Researchers. SAS Institute Inc, Cary (2002)

    Google Scholar 

  • Fan, X., Sivo, S.A.: Sensitivity of fit indices to model misspecification and model types. Multivar. Behav. Res. 42, 509–529 (2007)

    Article  Google Scholar 

  • Fan, X., Wang, L.: Effects of potential confounding factors on fit indices and parameter estimates for true and misspecified SEM models. Educ. Psychol. Meas. 58, 699–733 (1998)

    Google Scholar 

  • Ferrando, P.J.: Factor analytic procedures for assessing social desiderability in binary items. Multivar. Behav. Res. 40, 331–349 (2005)

    Article  Google Scholar 

  • Ferrando, P.J., Anguiano-Carrasco, C.: Assessing the impact of faking on binary personality measures: an IRT-based multiple-group factor analytic procedure. Multivar. Behav. Res. 44, 497–524 (2009)

    Article  Google Scholar 

  • Ferrando, P.J., Anguiano-Carrasco, C.: A structural modelbased optimal person-fit procedure for identifying faking. Educ. Psychol. Meas. 73, 173–190 (2013)

    Article  Google Scholar 

  • Flora, D.B., Curran, P.J.: An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychol. Methods 9, 466–491 (2004)

    Article  Google Scholar 

  • Forero, C.G., Maydeu-Olivares, A., Gallardo-Pujol, D.: Factor analysis with ordinal indicators: a Monte Carlo study comparing DWLS and ULS estimation. Struct. Equ. Model. 16, 625–641 (2009)

    Article  Google Scholar 

  • Fox, J.-P., Meijer, R.R.: Using item response theory to obtain individual information from randomized response data: an application using cheating data. Appl. Psychol. Meas. 32, 595–610 (2008)

    Article  Google Scholar 

  • Furnham, A.: Response bias, social desirability and dissimulation. Personal. Individ. Differ. 7, 385–400 (1986)

    Article  Google Scholar 

  • Gray, N.S., MacCulloch, M.J., Smith, J., Morris, M., Snowden, R.J.: Forensic psychology: violence viewed by psychopathic murderers. Nature 423, 497–498 (2003)

    Article  Google Scholar 

  • Helton, J.C., Johnson, J.D., Salaberry, C.J., Storlie, C.B.: Survey of sampling based methods for uncertainty and sensitivity analysis. Reliab. Eng. Syst. Saf. 91, 1175–1209 (2006)

    Article  Google Scholar 

  • Holden, R.R., Book, A.S.: Using hybrid Rasch-latent class modeling to improve the detection of fakers on a personality inventory. Personal. Individ. Differ. 47, 185–190 (2009)

    Article  Google Scholar 

  • Hopwood, C.J., Talbert, C.A., Morey, L.C., Rogers, R.: Testing the incremental utility of the negative impression-positive impression differential in detecting simulated personality assessment inventory profiles. J. Clin. Psychol. 64, 338–343 (2008)

    Article  Google Scholar 

  • Hu, L., Bentler, P.M.: Fit indices in covariance structure modeling: sensitivity to underparameterized model misspecification. Psychol. Methods 3, 424–453 (1998)

    Article  Google Scholar 

  • Jöreskog, K.: New developments in LISREL: analysis of ordinal variables using polychoric correlations and weighted least squares. Qual. & Quant. 24, 387–404 (1990)

    Article  Google Scholar 

  • Jöreskog, K., Sörbom, D.: LISREL V: analysis of Linear Structural Relationships by the Method of Maximum Likelihood. National Educational Resources, Chicago (1981)

    Google Scholar 

  • Jöreskog, K., Sörbom, D.: LISREL VI User’s Guide, 3rd edn. Scientific Software, Mooresville (1984)

    Google Scholar 

  • Jöreskog, K., Sörbom, D.: LISREL 8: user’s Reference Guide. Scientific Software International Inc, Lincolnwood (1996a)

    Google Scholar 

  • Jöreskog, K., Sörbom, D.: PRELIS 2: user’s Reference Guide. Scientific Software International Inc, Lincolnwood (1996b)

    Google Scholar 

  • Kenny, D.A., McCoach, D.B.: Effect of the number of variables on measures of fit in structural equation modeling. Struct. Equ. Model. 10, 333–351 (2003)

    Article  Google Scholar 

  • Lee, S.-Y., Poon, W.-Y., Bentler, P.M.: Full maximum likelihood analysis of structural equation models with polytomous variables. Stat. Probab. Lett. 9, 91–97 (1990)

    Article  Google Scholar 

  • Leite, W.L., Cooper, L.A.: Detecting social desiderability bias using factor mixture models. Multivar. Behav. Res. 45, 271–293 (2010)

    Article  Google Scholar 

  • Lombardi, L., Pastore, M.: Sensitivity of fit indices to fake perturbation of ordinal data: a sample by replacement approach. Multivar. Behav. Res. 47, 519–546 (2012)

    Article  Google Scholar 

  • Lombardi, L., Pastore, M.: sgr: a package for simulating conditional fake ordinal data. R. J. 6, 164–177 (2014)

    Google Scholar 

  • MacCann, C., Ziegler, M., Roberts, R.D.: Faking in personality assessment: Reflections and recommendations. New Perspect. Faking Personal. Assess., 309–329 (2011)

  • Marshall, E.: Scientific misconduct. How prevalent is fraud? That's a million-dollar question. Science. 290(5497), 1662–1663 (2000)

    Article  Google Scholar 

  • McCullagh, P., Nelder, J.A.: Generalized Linear Models. Chapman and Hall, London (1989)

    Book  Google Scholar 

  • McDonald, R.P., Marsh, H.W.: Choosing a multivariate model: Noncentrality and goodness of fit. Psychol. Bull. 107, 247–255 (1990)

    Article  Google Scholar 

  • McFarland, L.A., Ryan, A.M.: Variance in faking across noncognitive measures. J. Appl. Psychol. 85, 812–821 (2000)

    Article  Google Scholar 

  • Muthén, B.: A general structural equation model with dichotomous, ordered categorical and continuous latent variables indicators. Psychometrika 49, 115–132 (1984)

    Article  Google Scholar 

  • Muthén, B., Kaplan, D.: A comparison of some methodologies for the factor analysis of non-normal Likert variables: a note on the size of the model. Br. J. Math. Stat. Psychol. 45, 19–30 (1992)

    Article  Google Scholar 

  • Pastore, M., Lombardi, L.: The impact of faking on Cronbach’s alpha for dichotomous and ordered rating scores. Qual. & Quant. 48, 1191–1211 (2014)

    Article  Google Scholar 

  • Paulhus, D.L.: Two-component models of socially desirable responding. J. Personal. Soc. Psychol. 46, 598–609 (1984)

    Article  Google Scholar 

  • Paulhus, D.L.: Measurement and control of response bias. In: Robinson, J.P., Shaver, P.R., Wrightsman, L.S. (eds.) Measures of Personality and Socialpsychological Attitudes, pp. 17–59. Academic Press, New York (1991)

    Chapter  Google Scholar 

  • Paxton, P., Curran, P.J., Bollen, K.A., Kirby, J., Chen, F.: Monte Carlo experiments: design and implementation. Struct. Equ. Model. 8, 287–312 (2001)

    Article  Google Scholar 

  • Pek, J., MacCallum, R.C.: Sensitivity analysis in structural equation models: cases and their influence. Multivar. Behav. Res. 46, 202–228 (2011)

    Article  Google Scholar 

  • Ridgon, E.E., Ferguson, C.E.: The performance of the polychoric correlation coefficient and selected fitting functions in confirmatory factor analysis with ordinal data. J. Mark. Res. 28, 491–497 (1991)

    Article  Google Scholar 

  • Rosse, J.G., Stecher, M.D., Miller, J.L., Levin, R.A.: The impact of response distortion on preemployment personality testing and hiring decisions. J. Appl. Psychol. 83(4), 634–644 (1998)

    Article  Google Scholar 

  • Schermelleh-Engel, K., Moosbrugger, H., Mller, H.: Evaluating the fit of structural equation models: tests of significance and descriptive goodness- of-fit measures. Methods Psychol. Res. Online 8(2), 23–74 (2003)

    Google Scholar 

  • Steiger, J.H., Lind, J.C.: Statistically based tests for the number of common factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA (1980, May)

  • Tucker, L.R., Lewis, C.: A reliability coefficient for maximum likelihood factor analysis. Psychometrika 38, 1–10 (1973)

    Article  Google Scholar 

  • Van der Geest, S., Sarkodie, S.: The fake patient: a research experiment in a Ghanaian hospital. Soc. Sci. & Med. 47, 1373–1381 (1998)

    Article  Google Scholar 

  • Wood, S.N.: Generalized Additive Models. Taylor and Francis Group, Boca Raton (2006)

    Google Scholar 

  • Yang-Wallentin, F., Jöreskog, K., Luo, H.: Confirmatory factor analysis of ordinal variables with misspecified models. Struct. Equ. Model. 17, 392–423 (2010)

    Article  Google Scholar 

  • Zickar, M.J., Drasgow, F.: Detecting faking on a personality instrument using appropriateness measurement. Appl. Psychol. Meas. 20, 71–87 (1996)

    Article  Google Scholar 

  • Zickar, M.J., Robie, C.: Modeling faking good on personality items: an item-level analysis. J. Appl. Psychol. 84, 551–563 (1999)

    Article  Google Scholar 

  • Zickar, M.J., Gibby, R.E., Robie, C.: Uncovering faking samples in applicant, incumbent, and experimental data sets: an application of mixed-model item response theory. Organ. Res. Methods 7, 168–190 (2004)

    Article  Google Scholar 

  • Ziegler, M., Buehner, M.: Modeling socially desirable responding and its effects. Educ. & Psychol. Meas. 69, 548–565 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology and Cognitive Science, University of Trento, corso Bettini, 31, 38068, Rovereto, TN, Italy

    Luigi Lombardi

  2. Department of Developmental and Social Psychology, University of Padova, via Venezia, 8, 35131, Padova, Italy

    Massimiliano Pastore

Authors
  1. Luigi Lombardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Massimiliano Pastore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Massimiliano Pastore.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lombardi, L., Pastore, M. Robust evaluation of fit indices to fake-good perturbation of ordinal data. Qual Quant 50, 2651–2675 (2016). https://doi.org/10.1007/s11135-015-0282-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-015-0282-1

Keywords

Search

Navigation