Abstract
The proper estimation of age, period, and cohort (APC) effects is a pervasive concern for the study of a variety of psychological and social phenomena, inside and outside of organizations. One analytic technique that has been used to estimate APC effects is cross-temporal meta-analysis (CTMA). Although CTMA has some appealing qualities (e.g., ease of interpretability), it has also been criticized on theoretical and methodological grounds. Furthermore, CTMA makes strong assumptions about the nature and operation of cohort effects relative to age and period effects that have not been empirically tested. Accordingly, the goal of this paper is to explore CTMA, its history, and these assumptions. Using a Monte Carlo study, we demonstrate that, in many cases, cohort effects are misestimated (i.e., systematically over- or underestimated) by CTMA. This work provides further evidence that APC effects pose intractable problems for research questions where APC effects are of interest.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In Twenge’s (2000) paper, Twenge and Campbell (2000) is cited as an unpublished manuscript. Twenge and Campbell (2000) was later published in 2001 and hence appears as such in the “References” section. This explains the incongruity of citing a later paper (2001) that appeared in an earlier one (2000).
References
Baltes, P. B. (1968). Longitudinal and cross-sectional sequences in the study of age and generation effects. Human Development, 11, 145–171. https://doi.org/10.1159/000270604.
Baltes, P. B., Reese, H. W., & Lipsitt, L. P. (1980). Life-span developmental psychology. Annual Review of Psychology, 31(1), 65–110. https://doi.org/10.1146/annurev.ps.31.020180.000433.
Bandalos, D. L., & Gagné, P. (2012). Simulation methods in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 92–108). New York, NY, US: Guilford Press.
Bell, A., & Jones, K. (2013). The impossibility of separating age, period and cohort effects. Social Science & Medicine, 93, 163–165. https://doi.org/10.1016/j.socscimed.2013.04.029.
Bell, A., & Jones, K. (2014). Another 'futile quest'? A simulation study of Yang and Land's hierarchical age-period-cohort model. Demographic Research, 30, 333–360. https://doi.org/10.4054/DemRes.2013.30.11.
Benson, J., Brown, M., Glennie, M., O'Donnell, M., & O'Keefe, P. (2018). The generational “exchange” rate: How generations convert career development satisfaction into organisational commitment or neglect of work. Human Resource Management Journal, 28(4), 524–539. https://doi.org/10.1111/1748-8583.12198.
Bianchi, E. C. (2014). Entering adulthood in a recession tempers later narcissism. Psychological Science, 25, 1429–1437. https://doi.org/10.1177/0956797614532818.
Blyth, C. R. (1972). On Simpson's paradox and the sure-thing principle. Journal of the American Statistical Association, 67(338), 364–366. https://doi.org/10.1080/01621459.1972.10482387.
Bolger, N., & Laurenceau, J. P. (2013). Intensive longitudinal methods: An introduction to diary and experience sampling research. New York: Guilford Press.
Bubany, S. T., & Hansen, J. I. C. (2011). Birth cohort change in the vocational interests of female and male college students. Journal of Vocational Behavior, 78(1), 59–67. https://doi.org/10.1016/j.jvb.2010.08.002.
Campbell, S. M., Twenge, J. M., & Campbell, W. K. (2017). Fuzzy but useful constructs: Making sense of the differences between generations. Work, Aging and Retirement, 3(2), 130–139. https://doi.org/10.1093/workar/wax001.
Clark, D. M. T., Loxton, N. J., & Tobin, S. J. (2015). Declining loneliness over time: Evidence from American colleges and high schools. Personality and Social Psychology Bulletin, 41(1), 78–89. https://doi.org/10.1177/0146167214557007.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Lawrence Erlbaum.
Costanza, D. P., Darrow, J. B., Yost, A. B., & Severt, J. B. (2017). A review of analytical methods used to study generational differences: Strengths and limitations. Work, Aging and Retirement, 3(2), 149–165. https://doi.org/10.1093/workar/wax002.
Costanza, D. P., Darrow, J. B., Fraser, R. L., Severt, J. B., & Gade, P. A. (2012). Generational differences in work-related variables: A meta-analysis. Journal of Business and Psychology, 27(4), 375–394. https://doi.org/10.1007/s10869-012-9259-4.
Costanza, D. P., & Finkelstein, L. M. (2015). Generationally-based differences in the workplace: Is there a there there? Industrial and Organizational Psychology: Perspectives on Sciences and Practice, 8(3), 308–323. https://doi.org/10.1017/iop.2015.15.
Costanza, D. P., Finkelstein, L. M., Imose, R. A., & Ravid, D. M. in press Inappropriate inferences from generational research. In B. Hoffman, M. Shoss, & L. Wegman (Eds.), The Cambridge handbook of the changing nature of work. Cambridge, U.K.: Cambridge University Press.
Donnellan, M. B., Trzesniewski, K. H., & Robins, R. W. (2009). An emerging epidemic of narcissism or much ado about nothing? Journal of Research in Personality, 43(3), 498–501. https://doi.org/10.1016/j.jrp.2008.12.010.
Edershile, E. A., Woods, W. C., Sharpe, B. M., Crowe, M. L., Miller, J., & Wright, A. G. (2018, July 12). A day in the life of. Narcissus: Measuring narcissistic grandiosity and vulnerability in daily life. https://doi.org/10.31234/osf.io/jpqst.
Elder Jr., G. H. (1974). Children of the great depression: Social change in life experiences. Chicago, IL: University of Chicago Press.
Elder Jr., G. H., & Liker, J. K. (1982). Hard times in women’s lives: Historical in influences across forty years. American Journal of Sociology, 88, 241–269. https://doi.org/10.1086/227670.
Eschleman, K. J., King, M., Mast, D., Ornellas, R., & Hunter, D. (2017). The effects of stereotype activation on generational differences. Work, Aging and Retirement, 3(2), 200–208. https://doi.org/10.1093/workar/waw032.
Gentile, B., Wood, L. A., Twenge, J. M., Hoffman, B. J., & Campbell, W. K. (2015). The problem of generational change: Why cross-sectional designs are inadequate for investigating generational differences. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical myths and methodological urban legends. New York: Routledge.
Gergen, K. J. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26(2), 309–320. https://doi.org/10.1037/h0034436.
Gerstorf, D., Ram, N., Hoppmann, C., Willis, S. L., & Schaie, K. W. (2011). Cohort differences in cognitive aging and terminal decline in the Seattle Longitudinal Study. Developmental Psychology, 47, 1026–1041. https://doi.org/10.1037/a0023426.
Glenn, N. D. (1976). Cohort analysts' futile quest: Statistical attempts to separate age, period and cohort effects. American Sociological Review, 41(5), 900–904. https://doi.org/10.2307/2094738.
Glenn, N. D. (2005). Cohort analysis (2nd ed.). London: Sage.
Green, L., Fry, A. F., & Myerson, J. (1994). Discounting of delayed rewards: A life-span comparison. Psychological Science, 5(1), 33–36. https://doi.org/10.1111/j.1467-9280.1994.tb00610.x.
Hedges, L. V., & Becker, B. J. (1986). Statistical methods in the meta-analysis of research on gender differences. In J. S. Hyde & M. C. Linn (Eds.), The psychology of gender: Advances through meta-analysis (pp. 14–50). Baltimore, MD: Johns Hopkins University Press.
Hofer, S. M., & Sliwinski, M. J. (2006). Design and analysis of longitudinal studies on aging. In J. E. Birren & K. W. Schaie (Eds.), Handbook of the psychology of aging (6th ed., pp. 15–37). Amsterdam: Elsevier.
Huang, J. (2018). Changes of job burnout in Chinese nurses over 2004–2013: Cross-temporal meta analysis. Current Psychology, 37(3), 583–590. https://doi.org/10.1007/s12144-016-9540-1.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
Karazsia, B. T., Tylka, T. L., & Murnen, S. K. (2017). Is body dissatisfaction changing across time? A cross-temporal meta-analysis. Psychological Bulletin, 143(3), 293–320. https://doi.org/10.1037/bul0000081.
Klein, K. J., Dansereau, F., & Hall, R. J. (1994). Levels issues in theory development, data collection, and analysis. Academy of Management Review, 19, 195–229. https://doi.org/10.1177/109442810033001.
Kosloski, K. (1986). Isolating age, period, and cohort effects in developmental research: A critical review. Research on Aging, 8(4), 460–479. https://doi.org/10.1177/0164027586008004002.
Labouvie-Vief, G., Hakim-Larson, J., DeVoe, M., & Schoeberlein, S. (1989). Emotions and self-regulation: A life span view. Human Development, 32(5), 279–299. https://doi.org/10.1159/000276480.
Lub, X., Nije Bijvank, M., Bal, P. M., Blomme, R., & Schalk, R. (2012). Different or alike? Exploring the psychological contract and commitment of different generations of hospitality workers. International Journal of Contemporary Hospitality Management, 24(4), 553–573. https://doi.org/10.1108/09596111211226824.
Lyons, S. T., & Schweitzer, L. (2017). A qualitative exploration of generational identity: Making sense of young and old in the context of today’s workplace. Work, Aging and Retirement, 3(2), 209–224. https://doi.org/10.1093/workar/waw024.
Mackenzie, C., Erickson, J., Deane, F., & Wright, M. (2014). Changes in attitudes toward seeking mental health services: A 40-year cross-temporal meta-analysis. Clinical Psychology Review, 34(2), 99–106. https://doi.org/10.1016/j.cpr.2013.12.001.
Malahy, L. W., Rubinlicht, M. A., & Kaiser, C. R. (2009). Justifying inequality: A cross-temporal investigation of U.S. income disparities and just-world beliefs from 1973 to 2006. Social Justice Research, 22, 369–383. https://doi.org/10.1007/s11211-009-0103-6.
Ng, E. S., Johnson, J. M., & Burke, R. J. (2015). Millennials: Who are they, how are they different, and why should we care. In R. J. Burke, C. Cooper, & A.-S. Antoniou (Eds.), The multi-generational and aging workforce challenges and opportunities. Cheltenham, U.K.: Edward Elgar Publishing.
Ostroff, C. (1993). Comparing correlations based on individual-level and aggregated data. Journal of Applied Psychology, 78(4), 569–582. https://doi.org/10.1037/0021-9010.78.4.569.
Ostroff, C., & Harrison, D. A. (1999). Meta-analysis, level of analysis, and best estimates of population correlations: Cautions for interpreting meta-analytical results in organizational behavior. Journal of Applied Psychology, 84(2), 260–270. https://doi.org/10.1037/0021-9010.84.2.260.
Palmore, E. (1978). When can age, period, and cohort be separated? Social Forces, 57(1), 282–295. https://doi.org/10.2307/2577639.
Perry, E. L., Golom, F. D., Catenacci, L., Ingraham, M. E., Covais, E. M., & Molina, J. J. (2017). Talkin’ ‘bout your generation: The impact of applicant age and generation on hiring-related perceptions and outcomes. Work, Aging and Retirement, 3(2), 186–199. https://doi.org/10.1093/workar/waw029.
Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Pervasiveness of the IQ rise: A cross-temporal meta-analysis. PLoS ONE, 5(12), e14406. https://doi.org/10.1371/journal.pone.0014406.
Core Team, R. (2016). R: A language and environment for statistical computing. In R Foundation for Statistical Computing. Vienna: Austria. URL https://www.R-project.org/.
Roberts, B. W., Caspi, A., & Moffitt, T. E. (2001). The kids are alright: Growth and stability in personality development from adolescence to adulthood. Journal of Personality and Social Psychology, 81(4), 670–683. https://doi.org/10.1037/0022-3514.81.4.670.
Roberts, B. W., Walton, K. E., & Viechtbauer, W. (2006). Patterns of mean-level change in personality traits across the life course: A meta-analysis of longitudinal studies. Psychological Bulletin, 132(1), 1–25. https://doi.org/10.1037/0033-2909.132.1.1.
Roberts, B. W., & Wood, D. (2006). Personality development in the context of the neo-socioanalytic model of personality. In D. K. Mroczek & T. D. Little (Eds.), Handbook of personality development (pp. 11–39). Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers.
Robins, R. W., Fraley, R. C., Roberts, B. W., & Trzesniewski, K. H. (2001). A longitudinal study of personality change in young adulthood. Journal of Personality, 69(4), 617–640. https://doi.org/10.1037/0033-2909.132.1.1.
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15(3), 351–357. https://doi.org/10.2307/2087176.
Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82. https://doi.org/10.1146/annurev.psych.52.1.59.
Rosenthal, R., & Rosnow, R. L. (1984). Essentials of behavioral research: Methods and data analysis. New York: McGraw-Hill.
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in research: A correlational approach. New York: Cambridge University Press.
Rauvola, R. S., Rudolph, C. W., & Zacher, H. (2018). Generationalism: Problems and implications. Organizational Dynamics. https://doi.org/10.1016/j.orgdyn.2018.05.006.
Rudolph, C. W., Rauvola, R. S., & Zacher, H. (2018). Leadership and generations at work: A critical review. The Leadership Quarterly, 29(1), 44–57. https://doi.org/10.1016/j.leaqua.2017.09.004.
Rudolph, C. W. (2016). Lifespan developmental perspectives on working: A literature review of motivational theories. Work, Aging and Retirement, 2(2), 130–158. https://doi.org/10.1093/workar/waw012.
Rudolph, C. W., & Baltes, B. B. (2016). Age and health jointly moderate the influence of flexible work arrangements on work engagement: Evidence from two empirical studies. Journal of Occupational Health Psychology, 22(1), 40–58. https://doi.org/10.1037/a0040147.
Rudolph, C. W., & Zacher, H. (2015). Intergenerational perceptions and conflicts in multi-age and multigenerational work environments. In L. Finkelstein, D. Truxillo, F. Fraccaroli, & R. Kanfer (Eds.), SIOP organizational frontier series—Facing the challenges of a multi-age workforce: A use inspired approach (pp. 253–282). New York, NY: Psychology Press.
Rudolph, C. W., & Zacher, H. (2017). Considering generations from a lifespan developmental perspective. Work, Aging and Retirement, 3(2), 113–129. https://doi.org/10.1093/workar/waw019.
Rudolph, C. W., & Zacher, H. (2018). The kids are alright: Taking stock of generational differences at work. The Industrial-Organizational Psychologist, 55(3), 1–7.
Schaie, K. W. (1986). Beyond calendar definitions of age, time, and cohort: The general developmental model revisited. Developmental Review, 6(3), 252–277. https://doi.org/10.1016/0273-2297(86)90014-6.
Schaie, K. W. (1965). A general model for the study of developmental problems. Psychological Bulletin, 64(2), 92.
Schaie, K. W. (2013). Developmental in influences on adult intelligence: A Seattle longitudinal study (2nd ed.). New York, NY: Oxford University Press.
Srivastava, S., John, O. P., Gosling, S. D., & Potter, J. (2003). Development of personality in early and middle adulthood: Set like plaster or persistent change? Journal of Personality and Social Psychology, 84(5), 1041–1053. https://doi.org/10.1037/0022-3514.84.5.1041.
Stone, D. L., & Deadrick, D. L. (2015). Challenges and opportunities affecting the future of human resource management. Human Resource Management Review, 25(2), 139–145. https://doi.org/10.1016/j.hrmr.2015.01.003.
Szucs, D., & Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15(3), e2000797. https://doi.org/10.1371/journal.pbio.2000797.
Tan, J. Y., Huedo-Medina, T. B., Lennon, C. A., White, A. C., & Johnson, B. T. (2010). Us versus them in context: Meta-analysis as a tool for geotemporal trends in intergroup relations. International Journal of Conflict and Violence, 4(2), 288–297.
Trzesniewski, K. H., & Donnellan, M. B. (2010). Rethinking “generation me”: A study of cohort effects from 1976–2006. Perspectives on Psychological Science, 5(1), 58–75. https://doi.org/10.1177/1745691609356789.
Trzesniewski, K. H., Donnellan, M. B., & Robins, R. W. (2008). Is “generation me” really more narcissistic than previous generations? Journal of Personality, 76(4), 903–918. https://doi.org/10.1111/j.1467-6494.2008.00508.x.
Tufte, E. (1983). The visual display of quantitative information. Cheshire, Connecticut: Graphics Press.
Twenge, J. M. (1997a). Changes in masculine and feminine traits over time: A meta-analysis. Sex Roles, 36(5–6), 305–325. https://doi.org/10.1007/BF02766650.
Twenge, J. M. (1997b). Attitudes toward women, 1970–1995. Psychology of Women Quarterly, 21(1), 35–51. https://doi.org/10.1111/j.1471-6402.1997.tb00099.x.
Twenge, J. M. (2000). The age of anxiety? Birth cohort change in anxiety and neuroticism, 1952–1993. Journal of Personality and Social Psychology, 79(6), 1007–1021. https://doi.org/10.1037/0022-3514.79.6.1007.
Twenge, J. M. (2008). Generation me, the origins of birth cohort differences in personality traits, and cross-temporal meta-analysis. Social and Personality Psychology Compass, 2(3), 1440–1454. https://doi.org/10.1111/j.1751-9004.2008.00094.x.
Twenge, J. M. (2009). Generational changes and their impact in the classroom: Teaching Generation Me. Medical Education, 43(5), 398–405. https://doi.org/10.1111/j.1365-2923.2009.03310.x.
Twenge, J. M. (2017). IGen: Why today's super-connected kids are growing up less rebellious, more tolerant, less happy—and completely unprepared for adulthood—and what that means for the rest of us. New York: Simon and Schuster.
Twenge, J. M., & Campbell, W. K. (2001). Age and birth cohort differences in self-esteem: A cross-temporal meta-analysis. Personality and Social Psychology Review, 5(4), 321–344. https://doi.org/10.1207/S15327957PSPR05043.
Twenge, J. M., Campbell, S. M., Hoffman, B. J., & Lance, C. E. (2010). Generational differences in work values: Leisure and extrinsic values increasing, social and intrinsic values decreasing. Journal of Management, 36, 1117–1142. https://doi.org/10.1177/0149206309352246.
Twenge, J. M., Konrath, S., Foster, J. D., Campbell, W. K., & Bushman, B. J. (2008). Egos inflating over time: A cross-temporal meta-analysis of the Narcissistic Personality Inventory. Journal of Personality, 76(4), 875–902. https://doi.org/10.1111/j.1467-6494.2008.00507.x.
Twenge, J. M., Zhang, L., & Im, C. (2004). It’s beyond my control: A cross-temporal meta-analysis of increasing externality in Locus of Control, 1960–2002. Personality and Social Psychology Review, 8, 308–319. https://doi.org/10.1207/s15327957pspr08035.
Tymula, A., Belmaker, L. A. R., Ruderman, L., Glimcher, P. W., & Levy, I. (2013). Like cognitive function, decision making across the life span shows profound age-related changes. Proceedings of the National Academy of Sciences, 110(42), 17143–17148. https://doi.org/10.1073/pnas.1309909110.
Urick, M. J., Hollensbe, E. C., Masterson, S. S., & Lyons, S. T. (2017). Understanding and managing intergenerational conflict: An examination of influences and strategies. Work, Aging and Retirement, 3(2), 166–185. https://doi.org/10.1093/workar/waw009.
Walters, N. T., & Horton, R. (2015). A diary study of the influence of Facebook use on narcissism among male college students. Computers in Human Behavior, 52, 326–330. https://doi.org/10.1016/j.chb.2015.05.054.
Wegman, L. A., Hoffman, B. J., Carter, N. T., Twenge, J. M., & Guenole, N. (2018). Placing job characteristics in context: Cross-temporal meta-analysis of changes in job characteristics since 1975. Journal of Management, 44(1), 352–386. https://doi.org/10.1177/0149206316654545.
Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis (Vol. 59). Thousand Oaks, CA: Sage.
Yang, Y. (2008). Age, period, cohort effects. In D. Carr (Ed.), Encyclopedia of the life course and human development (Vol. 3, pp. 6–10). USA: Detroit: Macmillan Reference.
Yang, Z., Cao, F., Lu, H., Zhu, X., & Miao, D. (2014). Changes of anxiety in Chinese military personnel over time: A cross-temporal meta-analysis. International Journal of Mental Health Systems, 8(19), 1–9. https://doi.org/10.1186/1752-4458-8-19.
Yang, Y., & Land, K. C. (2006). A mixed models approach to the age-period-cohort analysis of repeated cross-section surveys, with an application to data on trends in verbal test scores. Sociological Methodology, 36(1), 75–97. https://doi.org/10.1111/j.1467-9531.2006.00175.x.
Yang, Y., & Land, K. C. (2013). Age-period-cohort analysis: New models, methods, and empirical applications. Boca Raton, FL: CRC Press.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
A pre-print version of this work can be found here: https://psyarxiv.com/exskp/. Code to replicate the simulations presented here can be found at: https://osf.io/mak6y/. A Shiny web-app is also available here: https://cortrudolph.shinyapps.io/CTMA_Simulation.
Rights and permissions
About this article
Cite this article
Rudolph, C.W., Costanza, D.P., Wright, C. et al. Cross-Temporal Meta-Analysis: A Conceptual and Empirical Critique. J Bus Psychol 35, 733–750 (2020). https://doi.org/10.1007/s10869-019-09659-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10869-019-09659-2