Design and Analysis

  • Joseph R. Scotti
  • Vanessa M. Jacoby
  • Stanley Cohen
  • Julie Hicks Patrick


A critical aspect of moving psychology into the realm of paradigmatic science, and of ensuring the development and meaningful dissemination of empirically supported interventions, is competency in research design and statistical analysis. We present evidence that psychology graduate programs are not performing as well as they might in this area of training. An expanded version of the Competency Cube Model is used to present the specific skills that are central to seven areas of research design and data analysis by levels of training (bachelors, master’s, doctoral) and career path (clinical, research). The seven areas include: problem conceptualization, measurement and scaling, research methodology, core statistics, specialized statistical procedures, data analysis and management, and communication of findings. In keeping with the Competency Cube Model, we discuss both foundational (broad professional skills) and functional (area-specific skills) competencies in the seven areas, and introduce three overlapping stages of competency, from familiarity to knowledge, knowledge to use, and use to contribution. An extended listing of skills by content area, level of training, and stage of competency is presented to expand the discussion on both the curricula that should be offered for various academic degrees, and the manner in which the knowledge and skills learned should be documented or demonstrated. We recognize the multiple variations and mixtures along a continuum from practicing clinical psychologist to basic research psychologist, and the resulting differences in competencies in research design and statistical analysis. As the field experiences a shortage of quantitative psychologists, researchers need to better document their competence in this area beyond the traditional means (e.g., peer-reviewed publications), as well as seek lifelong learning opportunities to expand their knowledge and skills. Clinical psychologists, on the other hand, must remain sufficiently conversant with this content area so as to evaluate published intervention research and competently evaluate their own practice.


Research Design Lifelong Learning Clinical Psychologist Career Path Reflective Practice 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. The following list includes some of our favorite items, both recent and “historical.” We note several items from the series Quantitative Applications in the Social Sciences, by Sage Publications, but recommend the entire series.Google Scholar
  2. Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of Ph.D. programs in North America. American Psy-chologist, 63, 32–50.CrossRefGoogle Scholar
  3. Aiken, L. S., West, S. G., Sechrest, L. B., & Reno, R. R. (1990). Graduate training in statistics, methodology, and measurement in psychology: An international survey. American Psychologist, 45, 721–734.CrossRefGoogle Scholar
  4. Aldenderfer, M. S., & Blashfield, R. K. (1984). Cluster analysis. In Sullivan, J. L., & Niemi, R. G., (Series Eds.), Quantitative Applications in the Social Sci-ences, 44. Beverly Hills, CA: Sage.Google Scholar
  5. American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.Google Scholar
  6. American Psychological Association. (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, 1060–1073.CrossRefGoogle Scholar
  7. American Psychological Association. (2005). National standards for high school psychology curricula, August 2005. Washington, DC: Author. Retrieved January 15, 2009, from
  8. American Psychological Association. (2007). APA guidelines for the undergraduate psychology major. Wash-ington, DC: Author. Retrieved January 15, 2009, from
  9. American Psychological Association. (2008a). Report of the task force for increasing the number of quantitative psychologists. Washington, DC: Author. Retrieved January 15, 2009, from
  10. American Psychological Association. (2008b). Teaching, learning, and assessing in a developmentally coherent curriculum. Washington, DC: American Psycholo-gical Association, Board of Educational Affairs. Retrieved January 15, 2009, from
  11. American Psychological Association. (2009). Job outlook for quantitative psychologists. Retrieved January 7, 2009, from
  12. American Psychological Association, Center for Work-force Studies. (n.d.). Table 4: Employment characteristics of APA members by membership status, 2000. Retrieved April 10, 2009, from
  13. Barlow, D. H., Hayes, S. C., & Nelson, R. O. (1984). The scientist practitioner: Research and accountability in clinical and educational settings. New York: Pergamon Press.Google Scholar
  14. Barlow, D. H., & Hersen, M. (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.). Elmsford, NY: Pergamon Press.Google Scholar
  15. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psy-chology, 51, 1173–1182.CrossRefGoogle Scholar
  16. Bechtoldt, H., Norcross, J. C., Wyckoff, L. A., Pokrywa, M. L., & Campbell, L. F. (2001). Theoretical orientations and employment settings of clinical and counseling psychologists: A comparative study. The Clinical Psychologist, 54(1), 3–6.Google Scholar
  17. Bernal, G., & Scharró-del-Río, M. (2001). Are empirically supported treatments valid for ethnic minorities? Toward an alternative approach for treatment research. Cultural Diversity and Ethnic Minority Psychology, 7, 328–342.PubMedCrossRefGoogle Scholar
  18. Bloom, M., Fischer, J., & Orme, J. G. (2009). Evaluating practice: Guidelines for the accountable professional (6th ed.). Boston: Pearson Higher Education.Google Scholar
  19. Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally College Publishing Co.Google Scholar
  20. Campion, M. A. (1993). Article review checklist: A criterion checklist for reviewing research articlesin applied psychology. Personnel Psychology, 46, 705–718.CrossRefGoogle Scholar
  21. Cleary, T. A. (1968). Test bias: Prediction of grades of negro and white students in integrated colleges. Journal of Educational Measurement, 5, 115–124.CrossRefGoogle Scholar
  22. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  23. Cohen, J. (1994). The Earth is round (p < .05). American Psychologist, 49, 997–1013.CrossRefGoogle Scholar
  24. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  25. Cohen, L. H. (1979). The research readership and information source reliance of clinical psychologists. Professional Psychology, 10, 780–785.CrossRefGoogle Scholar
  26. Council of Science Editors. (2009). Policy on journal access to scientific data. Retrieved January 31, 2009, from
  27. Council on Governmental Relations. (2006). Access to and retention of research data: Rights and responsibilities. Retrieved January 31, 2009, from
  28. Dana, R. H. (1993). Multicultural assessment perspectives for professional psychology. Boston: Allyn and Bacon.Google Scholar
  29. (n.d.). Competence. unabridged (v 1.1). Retrieved January 08, 2009, from
  30. Dillman, D. A. (2000). Mail and internet surveys: The tailored design method (2nd ed.). New York: Wiley.Google Scholar
  31. Elman, N. S., Illfelder-Kaye, J., & Robiner, W. N. (2005). Professional development: Training for professionalism as a foundation for competent practice in psychology. Professional Psychology: Research and Practice, 36, 367–375.CrossRefGoogle Scholar
  32. Field, A. (2005). Discovering statistics using SPSS (2nd ed.). London: Sage.Google Scholar
  33. Friedrich, J., Buday, E., & Kerr, D. (2000). Statistical training in psychology: A national survey and com-mentary on undergraduate programs. Teaching of Psychology, 27, 248–257.CrossRefGoogle Scholar
  34. Goldfried, M. R., & Wolfe, B. E. (1998). Toward a more clinically valid approach to therapy research. Journal of Consulting and Clinical Psychology, 66, 143–150.PubMedCrossRefGoogle Scholar
  35. Grimm, L. G., & Yarnold, P. R. (Eds.). (1995). Reading and understanding multivariate statistics. Washington, DC: American Psychological Association.Google Scholar
  36. Grimm, L. G., & Yarnold, P. R. (Eds.). (2000). Reading and understanding more multivariate statistics. Washing-ton, DC: American Psychological Association.Google Scholar
  37. Grissom, R. J., & Kim, J. J. (2005). Effect sizes for research: A broad practical approach. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  38. Heady, H. R. (2007). Rural veterans: A special concern for rural health advocates. Kansas City, MO: National Rural Health Association. Retrieved February 12, 2009, from
  39. Holmbeck, G. N. (1997). Toward terminological, conceptual, and statistical clarity in the study of mediators and moderators: Examples from the child-clinical and pediatric psychopathology literatures. Journal of Consulting and Clinical Psychology, 65, 599–610.PubMedCrossRefGoogle Scholar
  40. Huberty, C. J., & Morris, J. D. (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302–308.CrossRefGoogle Scholar
  41. Johnson, R. A., & Wichern, D. W. (1982). Applied multivariate statistical analysis. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  42. Johnston, J. M., & Pennypacker, H. S. (1980). Strategies and tactics of human behavioral research. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  43. Kaschak, M. P., & Moore, C. F. (2000). On the documentation of statistical analyses in the “clicky-box” era. American Psychologist, 55, 1511–1512.CrossRefGoogle Scholar
  44. Kaslow, N. J., Borden, K. A., Collins, F. L., Forrest, L., Illfelder-Kaye, J., Nelson, P. D. et al. (2004). Compe-tencies Conference: Future directions in education and credentialing in professional psychology. Journal of Clinical Psychology, 60, 699–712.PubMedCrossRefGoogle Scholar
  45. Kawachi, I., & Subramanian, S. V. (2006). Measuring and modeling the social and geographic context of trau-ma: A multilevel modeling approach. Journal of Tra-umatic Stress, 19, 195–203.CrossRefGoogle Scholar
  46. Kazdin, A. (1995). Preparing and evaluating research reports. Psychological Assessment, 7, 228–237.CrossRefGoogle Scholar
  47. Kazdin, A. E. (2003). Research design in clinical psychology (4th ed.). Boston: Allyn and Bacon.Google Scholar
  48. Keane, T. M., Fairbank, J. A., Caddell, J. M., Zimering, R.T., & Bender, M. E. (1985). A behavioral approach to assessing and treating post-traumatic stress disorder in Vietnam veterans. In Figley, C. R. (Ed.), Trauma and its wake: The study and treatment of post-traumatic stress disorder (pp. 257-294). New York: Brunner/Mazel.Google Scholar
  49. Kim, J., & Mueller, C. W. (1978). Introduction to factor analysis: What it is and how to do it. In Sullivan, J. L., & Niemi, R. G. (Series Eds.), Quantitative Applications in the Social Sciences, 13. Beverly Hills, CA: Sage.Google Scholar
  50. Kuhn, T. S. (1996). The structure of scientific revolutions (3rd ed.). Chicago: University of Chicago Press.Google Scholar
  51. Landau, S. (2002). Using survival analysis in psychology. Understanding Statistics, 1, 233–270.CrossRefGoogle Scholar
  52. Leigh, I W., Smith, I. L., Bebeau, M. J., Lichtenberg, J. W., Nelson, P. D., Portnoy, S. et al. (2007). Competency assessment models. Professional Psychology: Research and Practice, 38, 463–473.CrossRefGoogle Scholar
  53. Lichtenberg, J. W., Portnoy, S., Bebeau, M. J., Leigh, I.W., Nelson, P. D., Rubin, N. J. et al. (2007). Challenges to the assessment of competence and competencies. Professional Psychology: Research and Practice, 38, 474–478.CrossRefGoogle Scholar
  54. Lodge, M. (1981). Magnitude scaling: Quantitative measurement of opinions. In Sullivan, J. L, & Niemi, R. G. (Series Eds.), Quantitative Applications in the Social Sciences, 25. Beverly Hills, CA: Sage.Google Scholar
  55. Marshall, C., & Rossman, C. B. (1989). Designing qualitative research. Newbury Park, CA: Sage.Google Scholar
  56. MacDonald, R. P., & Ho, M-H. R. (2002). Principles and practices in reporting structural equation analyses. Psychological Methods, 7, 64–82.CrossRefGoogle Scholar
  57. McBurney, D. H. (2001). Research methods (5th ed.). Belmont, CA: Wadsworth Thomas Learning.Google Scholar
  58. McNemar, Q. (1962). Psychological statistics (3rd ed.). New York: Wiley.Google Scholar
  59. Mooney, C. Z., & Duval, R. D. (1993). Bootstrapping: A nonparametric approach to statistical inference. In Lewis-Beck, M. S. (Series Ed.), Quantitative Applications in the Social Sciences, 95. Newbury Park, CA: Sage.Google Scholar
  60. Murphy, K. R., & Myors, B. (2004). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  61. Nathan, P. E. (2000). The Boulder model: A dream deferred – or lost? American Psychologist, 55, 250–252.PubMedCrossRefGoogle Scholar
  62. National Association of School Psychologists. (2003). NCSP renewal guidelines. Retrieved May 1, 2009, from
  63. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.Google Scholar
  64. Office of Management and Budget. (2003). Department of Veterans Affairs. Retrieved April 10, 2009, from
  65. Pedhazur, E. J. (1982). Multiple regression in behavioral research: Explanation and prediction (2nd ed.). New York: CBS College Publishing.Google Scholar
  66. Persons, J. B., & Silberschatz, G. (1998). Are results of randomized controlled trials useful to psychotherapists? Journal of Consulting and Clinical Psychology, 66, 126–135.PubMedCrossRefGoogle Scholar
  67. Popper, K. (2002a). Conjectures and refutations: The growth of scientific knowledge. New York: Routledge. (Original work published 1963)Google Scholar
  68. Popper, K. (2002b). The logic of scientific discovery NewYork: Routledge. (Original work published 1935)Google Scholar
  69. Radloff, L. S. (1977). The CES-D scale: A self report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401.CrossRefGoogle Scholar
  70. Reilly, D. H., Barclay, J., & Culbertson, F. (1977). The current status of competency-based training, Pt. 1: Validity, reliability, logistical, and ethical issues. Journal of School Psychology, 34, 474–475.Google Scholar
  71. Roberts, M. C., Borden, K. A., Christiansen, M. D., & Lopez, S. J. (2005). Fostering a culture shift: Assessment of competence in the education and careers of professional psychologists. Professional Psychology: Research and Practice, 36, 355–361.CrossRefGoogle Scholar
  72. Rodolfa, E., Bent, R., Eisman, E., Nelson, P., Rehm, L., & Ritchie, P. (2005). A cube model for competency development: Implications for psychology educators and regulators. Professional Psychology: Research and Practice, 36, 347–354.CrossRefGoogle Scholar
  73. Rosenthal, R. (1984). Meta-analytic procedures for social research. In Bickman, L., & Rog, D. (Series Eds.), Applied Social Research Methods, 6. Beverly Hills, CA: Sage.Google Scholar
  74. Rosselló, J., Bernal, G., & Rivera-Medina, C. (2008). Individual and group CBT and IPT for Puerto Rican adolescents with depressive symptoms. Cultural Diversity and Ethnic Minority Psychology, 14, 234–245.PubMedCrossRefGoogle Scholar
  75. Rubin, D. B. (1987). Multiple imputations for nonresponse in surveys. New York: Wiley.CrossRefGoogle Scholar
  76. Ruggiero, K. J., Del Ben, K. S., Scotti, J. R., & Rabalais, A. E. (2003). Psychometric properties of the PTSD checklist-civilian version. Journal of Traumatic Stress, 16, 495–502.PubMedCrossRefGoogle Scholar
  77. Sattler, J. M. (1988). Assessment of children (3rd ed.). San Diego: Sattler.Google Scholar
  78. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 142–177.CrossRefGoogle Scholar
  79. Scotti, J. R., Majewski, V., Heady, H., & Tunick, R. (2008, November). West Virginia veterans returning from Iraq and Afghanistan: Impact on personal and family functioning. In Scotti, J. R. (Chair; J. Lyons, Discussant), Addressing barriers to service utilization for returning Iraq and Afghanistan veterans and families. Symposium and paper presented at the annual meeting of the International Society for Traumatic Stress Studies, Chicago.Google Scholar
  80. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company.Google Scholar
  81. Sidman, M. (1960). Tactics of scientific research: Evaluating experimental data in psychology. New York: Basic Books.Google Scholar
  82. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford: Oxford University Press.Google Scholar
  83. Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.Google Scholar
  84. Sonis, J., Palmieri, P. A., Lauterbach, D., King, L. A., & King, D. W. (2008). Introduction to the special section: Innovations in trauma research methods, 2007. Journal of Traumatic Stress, 21, 431–432.PubMedCrossRefGoogle Scholar
  85. Staats, A. W. (1990). Paradigmatic behavior therapy: A unified framework for theory, research, and practice. In Eifert, G. H., & Evans, I. M. (Eds.), Unifying behavioral therapy: Contributions of paradigmatic behaviorism (pp. 14-54). New York: Springer.Google Scholar
  86. Staats, A. W. (1995). Paradigmatic behaviorism and paradigmatic behavior therapy. In O’Donohue, W., & Krasner, L. (Eds.), Theories of behavior therapy: Exploring behavior change (pp. 659-693). Washing-ton, DC: American Psychological Association.Google Scholar
  87. Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York: Harper Collins College.Google Scholar
  88. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Pearson Education.Google Scholar
  89. Tanielian, T., & Jaycox, L. H. (Eds.). (2008). Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery. Santa Monica, CA: RAND Corporation.Google Scholar
  90. Thomas, J. C., & Hersen, M. (Eds.). (2003). Understand-ing research in clinical and counseling psychology. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  91. Tomcho, T. J., Rice, D., Foels, R., Folmsbee, L., Vladescu, J., Lissman, R. et al. (2009). APA’s learning objectives for research methods and statistics in practice: A multimethod analysis. Teaching of Psychology, 36, 84–89.CrossRefGoogle Scholar
  92. Viswanathan, M. (2005). Measurement error and research design. Thousand Oaks, CA: Sage.Google Scholar
  93. Watkins, Jr., C. E., Schneider, L. J., Manus, M., & Hunton-Shoup, J. (1990). Terminal master’s-level training in counseling psychology: Skills, competencies, and student interests. Professional Psychology: Research and Practice, 21, 216–218.CrossRefGoogle Scholar
  94. Weathers, F., Litz, B., Herman, D., Huska, J., & Keane, T. (1993, October). The PTSD Checklist (PCL): Reliability, validity, and diagnostic utility. Paper presented at the annual convention of the International Society for Traumatic Stress Studies, San Antonio, TX.Google Scholar
  95. Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychological journals: Guidelines and explanations. American Psychologist, 54, 594–604.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media LLC 2010

Authors and Affiliations

  • Joseph R. Scotti
    • 1
  • Vanessa M. Jacoby
    • 1
  • Stanley Cohen
    • 1
  • Julie Hicks Patrick
    • 1
  1. 1.West Virginia UniversityMorgantownUSA

Personalised recommendations