, Volume 63, Issue 4, pp 315–340 | Cite as

Data, model, conclusion, doing it again

  • Ivo W. Molenaar


This paper explores the robustness of conclusions from a statistical model against variations in model choice (rather than variations in random sampling and random assignment to treatments, which are the usual variations covered by inferential statistics). After the problem formulation in section 1, section 2 presents an example from Box and Tiao in which variation in parent distribution is modeled for a one sample location problem. An adaptive Bayesian procedure permits to use a weighted mixture of parent distributions rather than choosing just one, such as a normal or a uniform distribution.

In section 3 similar considerations are applied to an event history model for the influence of education and gender on age at first marriage, but here the conclusions are hardly influenced by the choice of the duration distribution. In section 4 a brief discussion of model choice in factor analysis and structural equation models is followed by a more elaborate treatment of the choice of integer valued slopes for item response functions in the OPLM model which is an extension of the Rasch model. A modest simulation study suggests that Adaptive Bayesian Modeling with a mixture of sets of slopes works better than fixing one set of postulated slopes.

The paper concludes with some remarks on the roles and sources of prior distributions followed by a short epilogue which argues that simultaneous consideration of a class of models for the same data is sometimes superior to exclusively analyzing the data under one specific model chosen from such a class.

Key words

adaptive Bayesian modeling model choice posterior model weight robustness under model choice one parameter logistic model 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bernardo, J. M., & Smith, A. F. M. (1994).Bayesian theory. Chichester: John Wiley & Sons.Google Scholar
  2. Blossfeld, H-P., & Rohwer, G. (1995).Techniques of event history modeling. New approaches to causal analysis. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  3. Boomsma, A. (1991).BOJA. A program for bootstrap and jackknife analysis (User's manual, PC version). Groningen: iec ProGAMMA.Google Scholar
  4. Box, G. E. P., & Andersen, S. L. (1955). Permutation theory in the derivation of robust criteria and the study of departures from assumptions.Journal of the Royal Statistical Society, Series B, 17.Google Scholar
  5. Box, G. E. P., & Tiao, G. C. (1973).Bayesian inference in statistical analysis. Menlo Park, CA: Addison-Wesley. (Subsequently published by John Wiley & Sons)Google Scholar
  6. Bozdogan, H. (1987). Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions.Psychometrika, 52, 345–370.CrossRefGoogle Scholar
  7. Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures.British Journal of Mathematical and Statistical Psychology, 37, 62–83.PubMedGoogle Scholar
  8. Camstra, A. (1998).Cross-validation in covariance structure analysis. Unpublished doctoral dissertation. University of Groningen, The Netherlands.Google Scholar
  9. Cattell, R. B. (1966). The scree test for the number of factors.Multivariate Behavioral Research, 1, 245–276.Google Scholar
  10. Cliff, N. (1979). Test theory without true scores?Psychometrika, 44, 373–393.CrossRefGoogle Scholar
  11. Cox, D. R. (1972). Regression models and life-tables.Journal of the Royal Statistical Society, Series B,34, 187–220.Google Scholar
  12. Cudeck, R., & Browne, M. W. (1983). Cross-validation for prediction.Multivariate Behavioral Research, 18, 147–167.CrossRefGoogle Scholar
  13. De Gooijer, J. G. (1995). Cross-validation criteria for covariance structures.Communications in Statistics: Simulation and Computation, 24, 1–16.Google Scholar
  14. De Groot, A. D. (1984). The theory of the science forum: Subject and purport.Methodology and Science, 17, 230–259.Google Scholar
  15. de Leeuw, J. (1988). Models and techniques.Statistica Neerlandica, 42, 91–98.Google Scholar
  16. Ellis, J. L., & Junker, B. W. (1997). Tail-measurability in monotone latent variable models,Psychometrika, 62, 495–523.CrossRefGoogle Scholar
  17. Fisher, R. A. (1960).The design of experiments (7th ed.). New York: Hafner.Google Scholar
  18. Fischer, G. H. (1974).Einführung in die Theorie Psychologischer Tests. [Introduction to mental test theory.] Bern: Huber.Google Scholar
  19. Fischer, G. H., & Molenaar, I. W. (Eds.). (1995).Rasch models: Foundations, Recent developments, and applications. New York: Springer.Google Scholar
  20. Glas, C. A. W. (1988). The derivation of some tests for the Rasch model from the multinomial distribution.Psychometrika, 53, 525–546.CrossRefGoogle Scholar
  21. Hoijtink, H., & Molenaar, I. W. (1997). A multidimensional item response model: Constrained latent class analysis using the Gibbs sampler and posterior predictive checks.Psychometrika, 62, 171–189.Google Scholar
  22. Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling,Sociological Methods & Research, 26, 329–367.Google Scholar
  23. Junker, B. W., & Ellis, J. L. (1997). A characterization of monotone unidimensional latent variable models,Annals of Statistics, 25, 1327–1343.Google Scholar
  24. Kelley, T. L. (1927).Fundamentals of statistics. Cambridge, MA: Harvard University Press.Google Scholar
  25. Luijben, T. C. W. (1989).Statistical guidance for model modification in covariance structure analysis. Amsterdam: Sociometric research Foundation.Google Scholar
  26. Mokken, R. J. (1971).A theory and procedure of scale analysis. The Hague: Mouton/Berlin: De Gruyter.Google Scholar
  27. Molenaar, I. W. (1988). Formal statistics and informal data analysis or why laziness should be discouraged.Statistica Neerlandica, 42, 83–90.Google Scholar
  28. Molenaar, I. W. (1997). [Review ofTechniques of Event History Modeling. New Approaches to Causal Analysis by Hans-Peter Blossfeld & Goetz Rohwer].Psychometrika, 62, 461–462.Google Scholar
  29. Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference, Part I,Biometrika, Series A,20, 175.Google Scholar
  30. Novick, M. R., Jackson, P. H., & Thayer, D. T. (1971). Bayesian inference and the classical test theory model: reliability and true scores.Psychometrika, 36, 261–288.Google Scholar
  31. Novick, M. R., Lewis, C., & Jackson, P. H. (1973). The estimation of proportions in m groups.Psychometrika, 38, 19–46.CrossRefGoogle Scholar
  32. Patel, N., & Mehta, C. R. (1989).StatXact user manual. Cambridge, MA: Cytel Software.Google Scholar
  33. Raftery, A. (1998).How many clusters? Keynote Address presented at the Joint Meeting of Classification Society of North America and Psychometric Society, Champaign IL.Google Scholar
  34. Ramsay, J. O. (1996). A similarity based smoothing approach to nondimensional item analysis.Psychometrika, 60, 323–339.Google Scholar
  35. Ramsay, J. O., & Abramowicz, M. (1989). Binomial regression with monotone splines: A psychometric application.Journal of the American Statistical Association, 84, 906–915.Google Scholar
  36. Rasch, G. (1960).Probabilistic models for some intelligence and attainment tests. Copenhagen: The Danish Institute of Educational Research. (Expanded edition published in 1980, Chicago: The University of Chicago Press)Google Scholar
  37. Saris, W. E., Satorra, A., & Sörbom, D. (1987). The detection and correction of specification errors in structural equation models. In C. C. Clogg (Ed.),Sociological methodology (pp. 105–129). Washington, DC: American Sociological Association.Google Scholar
  38. Sijtsma, K., & Molenaar, I. W. (1999).A nonparametric item response theory. Submitted for publication.Google Scholar
  39. Smeenk, W. H. (in press).Opportunity and marriage; the impact of individual resources and marriage market structure on the when and whom of first marriage in the Netherlands. Amsterdam: Thesis Publishers.Google Scholar
  40. Sörbom, D. (1989). Model modification.Psychometrika, 54, 371–384.CrossRefGoogle Scholar
  41. Stout, W. F. (1987). A nonparametric approach for assessing latent trait unidimensionality.Psychometrika, 52, 589–617.CrossRefGoogle Scholar
  42. Stout, W. F. (1990). A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation.Psychometrika, 55, 293–325.Google Scholar
  43. Swaminathan, H., & Gifford, J. A. (1985). Bayesian estimation in the two-parameter logistic model.Psychometrika, 50, 349–364.CrossRefGoogle Scholar
  44. Thissen, D., & Steinberg, L. (1984). A response model for multiple choice items,Psychometrika, 49, 501–519.Google Scholar
  45. Ultee, W. C., & Ganzeboom, H. B. G. (1993).Netherlands Family Survey 1992–1993 [machine readable data file]. Nijmegen, Netherlands: Nijmegen University, Department of Sociology.Google Scholar
  46. Van Lenthe, J. (1993). ELI: An interactive elicitation technique for subjective probability distributions.Organizational Behavior and Human Decision Processes, 55, 379–413.Google Scholar
  47. Van Lenthe, J. (1994). Scoring-rule feedforward and the elicitation of subjective probability distributions.Organizational Behavior and Human Decision Processes, 59, 188–209.Google Scholar
  48. Verhelst, N. D., & Glas, C. A. W. (1995). The one parameter logistic model. In G. H. Fischer & I. W. Molenaar (Eds.),Rasch models: Foundations, recent developments and applications (pp. 215–237). New York: Springer.Google Scholar
  49. Warm, Th. A. (1989). Weighted likelihood estimation of ability in item response theory.Psychometrika, 54, 427–450.CrossRefGoogle Scholar
  50. Wright, G., & Ayton, P. (1994).Subjective probability. New York: John Wiley and Sons.Google Scholar

Copyright information

© The Psychometric Society 1998

Authors and Affiliations

  • Ivo W. Molenaar
    • 1
  1. 1.University of GroningenThe Netherlands

Personalised recommendations