Skip to main content

Meta-Analysis for Nonprofit Research: Synthesizing Quantitative Evidence for Knowledge Advancement

Abstract

The past two decades have witnessed massive growth in the amount of quantitative research in nonprofit studies. Despite the large number of studies, findings from these studies have not always been consistent and cumulative. The diverse and competing findings constitute a barrier to offering clear, coherent knowledge for both research and practice. To further advance nonprofit studies, some have called for meta-analysis to synthesize inconsistent findings. Although meta-analysis has been increasingly used in nonprofit studies in the past decade, many researchers are still not familiar with the method. This article thus introduces meta-analysis to nonprofit scholars and, through an example demonstration, provides general guidelines for nonprofit scholars with background in statistical methods to conduct meta-analyses, with a focus on various judgement calls throughout the research process. This article could help nonprofit scholars who are interested in using meta-analysis to address some unsolved research questions in the nonprofit literature.

This is a preview of subscription content, access via your institution.

Fig. 1

Notes

  1. In this manuscript, we use nonprofit studies to refer to the studies on voluntary actions, nonprofit organizations, and civil society.

  2. We also searched some other related journals in nonprofit studies including International Review on Public and Nonprofit Marketing, Journal of Nonprofit & Public Sector Marketing, Journal of Philanthropy and Marketing, Journal of Nonprofit Education and Leadership, Nonprofit Policy Forum, and only found one meta-analysis (Xu & Huang, 2020).

  3. This article introduces the basic steps and important judgment calls for nonprofit scholars who are interested in conducting meta-analysis. The discussion in this section is not exhaustive. Interested readers should refer to meta-analysis textbooks listed in the references.

  4. Ringquist (2013, pp. 121 to 124) provides detailed information about choosing appropriate statistics to estimate effect sizes.

  5. Before analyzing effect sizes, researchers need to choose between a fixed effects and a random effects framework. There are two approaches that researchers can use to decide which framework to use. They are the Q test and I2 statistic approaches. Researchers can use the Q test approach to identify excess variance in a sample of effect sizes, and use the I2 statistic approach to assess the magnitude of the variability in effect sizes that is not attributable to sampling errors. Ringquist (2013, pp. 121 to 124) offers detailed information to conduct the Q test and calculate I2 statistic, providing criteria to choose between the two frameworks. Generally, random effects models are more widely used in social science, since it is less reasonable to assume that all studies shares the same, one common effect.

  6. The inverse variance weight is a function of sample size. Effect sizes from studies that have larger sample are placed more weight.

  7. There might be effects that lie an abnormal distance from other effects. They are outliers. We suggest that researchers exclude them before combing effect size or conduct robustness checks by reporting average effect sizes with and without those outliers.

References

  • Blom, R., Kruyen, P. M., Van der Heijden, B. I., & Van Thiel, S. (2020). One HRM fits all? A meta-analysis of the effects of HRM practices in the public, semipublic, and private sector. Review of Public Personnel Administration, 40(1), 3–35.

    Article  Google Scholar 

  • Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–236). Russell Sage.

    Google Scholar 

  • Cheung, M. W. L. (2015). Meta-analysis: A structural equation modeling approach. John Wiley & Sons.

    Google Scholar 

  • Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12–37.

    Article  Google Scholar 

  • Chapman, C. M., Hornsey, M. J., & Gillespie, N. (2021). To what extent is trust a prerequisite for charitable giving? A systematic review and meta-analysis. Nonprofit and Voluntary Sector Quarterly, 50, 1274–1303. https://doi.org/10.1177/08997640211003250

    Article  Google Scholar 

  • Cooper, H. (2017). Research synthesis and meta-analysis: A step-by-step approach. Sage.

    Book  Google Scholar 

  • Daniel, J. L., & Kim, M. (2018). The scale of mission-embeddedness as a nonprofit revenue classification tool: Different earned revenue types, different performance effects. Administration & Society, 50(7), 947–972.

    Article  Google Scholar 

  • De Wit, A., & Bekkers, R. (2017). Government support and charitable donations: A meta-analysis of the crowding-out hypothesis. Journal of Public Administration Research and Theory, 27(2), 301–319.

    Google Scholar 

  • Fleiss, J. L., & Berlin, J. A. (2009). Effect sizes for dichotomous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 237–254). Russell Sage.

    Google Scholar 

  • Geyskens, I., Krishnan, R., Steenkamp, J. B. E., & Cunha, P. V. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35(2), 393–419.

    Article  Google Scholar 

  • Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8.

    Article  Google Scholar 

  • Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018). Meta-analysis and the science of research synthesis. Nature, 555(7695), 175–182.

    Article  Google Scholar 

  • Hedges, L., & Olkin, I. (1985). Statistical methods for meta-analysis. Academic Press.

    Google Scholar 

  • Hung, C. (2020). Commercialization and nonprofit donations: A meta-analytic assessment and extension. Nonprofit Management and Leadership. https://doi.org/10.1002/nml.21435

    Article  Google Scholar 

  • Hung, C., & Hager, M. A. (2019). The impact of revenue diversification on nonprofit financial health: A meta-analysis. Nonprofit and Voluntary Sector Quarterly, 48(1), 5–27.

    Article  Google Scholar 

  • Hunt, M. (1997). How science takes stock: The story of meta-analysis. Russell Sage Foundation.

    Google Scholar 

  • Jackson, S. K., Guerrero, S., & Appe, S. (2014). The state of nonprofit and philanthropic studies doctoral education. Nonprofit and Voluntary Sector Quarterly, 43(5), 795–811.

    Article  Google Scholar 

  • Kim, M. (2017). The relationship of nonprofits’ financial health to program outcomes: Empirical evidence from nonprofit arts organizations. Nonprofit and Voluntary Sector Quarterly, 46(3), 525–548.

    Article  Google Scholar 

  • Kulik, J. A., & Kulik, C. L. C. (1989). Meta-analysis in education. International Journal of Educational Research, 13(3), 221–340.

    Article  Google Scholar 

  • Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Sage.

    Google Scholar 

  • Lu, J. (2016). The philanthropic consequence of government grants to nonprofit organizations: A meta-analysis. Nonprofit Management and Leadership, 26(4), 381–400.

    Article  Google Scholar 

  • Lu, J. (2017). Does population heterogeneity really matter to nonprofit sector size? Revisiting Weisbrod’s demand heterogeneity hypothesis. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 1–27.

  • Lu, J. (2018). Organizational antecedents of nonprofit engagement in policy advocacy: A meta-analytical review. Nonprofit and Voluntary Sector Quarterly, 47(4_suppl), 177S-203S.

    Article  Google Scholar 

  • Lu, J., & Xu, C. (2018). Complementary or supplementary? The relationship between government size and nonprofit sector size. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 29(3), 454–469.

    Article  Google Scholar 

  • Lu, J., Lin, W., & Wang, Q. (2019). Does a more diversified revenue structure lead to greater financial capacity and less vulnerability in nonprofit organizations? A bibliometric and meta-analysis. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 30(3), 593–609.

    Article  Google Scholar 

  • Ma, J., & Konrath, S. (2018). A century of nonprofit studies: Scaling the knowledge of the field. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 29(6), 1139–1158.

    Article  Google Scholar 

  • Pearson, K. (1904). Report on certain enteric fever inoculation statistics. BMJ, 3, 1243–1246.

    Google Scholar 

  • Pfeffer, J. (1993). Barriers to the advance of organizational science: Paradigm development as a dependent variable. Academy of Management Review, 18(4), 599–620.

    Article  Google Scholar 

  • Reed, J. G., & Baxter, P. M. (2009). Using reference databases. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 73–101). Russell Sage Foundation.

    Google Scholar 

  • Ringquist, E. (2013). Meta-analysis for public management and policy. John Wiley & Sons.

    Google Scholar 

  • Rosethal, R., & DiMatteo, M. (2001). Meta-analysis: Recent developments in quantitative methods for literature review. Annual Review of Psychology, 52, 59–82.

    Article  Google Scholar 

  • Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2006). Publication bias in meta-analysis: Prevention, assessment and adjustments. John Wiley & Sons.

    Google Scholar 

  • Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173–1181.

    Article  Google Scholar 

  • Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings. Sage.

    Book  Google Scholar 

  • Shadish, W. R., & Lecy, J. D. (2015). The meta-analytic big bang. Research Synthesis Methods, 6(3), 246–264.

    Article  Google Scholar 

  • Shoham, A., Ruvio, A., Vigoda-Gadot, E., & Schwabsky, N. (2006). Market orientations in the nonprofit and voluntary sector: A meta-analysis of their relationships with organizational performance. Nonprofit and Voluntary Sector Quarterly, 35(3), 453–476.

    Article  Google Scholar 

  • Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752–760.

    Article  Google Scholar 

  • Stanley, T. D., & Jarrell, S. B. (2005). Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 19(3), 299–308.

    Article  Google Scholar 

  • Sutton, A. J. (2009). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435–452). Russell Sage Foundation.

    Google Scholar 

  • Sutton, A. J., Abrams, K. R., Jones, D. R., Jones, D. R., Sheldon, T. A., & Song, F. (2000). Methods for meta-analysis in medical research. Wiley.

    Google Scholar 

  • Thompson, S. G., & Higgins, J. P. (2002). How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21(11), 1559–1573.

    Article  Google Scholar 

  • Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10(2), 161–179.

    Article  Google Scholar 

  • Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222.

    Article  Google Scholar 

  • Xu, J., & Huang, G. (2020). The relative effectiveness of gain-framed and loss-framed messages in charity advertising: Meta-analytic evidence and implications. International Journal of Nonprofit and Voluntary Sector Marketing, 25(4), e1675.

    Article  Google Scholar 

  • Willems, J., Boenigk, S., & Jegers, M. (2014). Seven trade-offs in measuring nonprofit performance and effectiveness. Voluntas: International Journal of Voluntary and Nonprofit Organizations, 25(6), 1648–1670.

    Article  Google Scholar 

Download references

Funding

This study was not funded by any organization.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to ChiaKo Hung.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interests.

Human and Animal Rights

This study does not involve Human Participants and/or Animals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix 1. Published Meta-Analyses in Three Leading Nonprofit Studies Journals

Appendix 1. Published Meta-Analyses in Three Leading Nonprofit Studies Journals

Study Journal Dependent variable Focal predictors # studies included # effect sizes analyzed Moderators Main findings
Shoham et al. (2006) NVSQ Organizational performance Market orientation 11 11 Country location (USA versus other nations), market orientation measure (behavioral versus philosophical measure), performance measure (subjective versus objective measures) There is a positive association between the two variables
The relationship is stronger in nonprofits than for-profits
Country location and market orientation measure moderate the relationship
Lu (2016) NML Private donations Government grants 60 637 Nonprofit subsector, country location (USA versus other nations), control for age, control for size, data structure (longitudinal versus others), lagged effect, endogeneity correction There is a small, positive correlation between the two variables
Nonprofit subsector, control for age, data structure, and endogeneity moderate the relationship
Lu (2017) Voluntas Nonprofit sector size Population heterogeneity 37 491 Country location (USA versus other nations, within-country versus cross-country), nonprofit sector size measure (density, finance, employment, etc.), population heterogeneity measure (age, employment status, ethnicity, gender, etc.) There is a small, positive association between the two variables
The relationship is generalized across countries and measurements of nonprofit sector size
Population heterogeneity in terms of age, education, ethnicity, language, and religion predicts nonprofit sector size better
Lu (2018) NVSQ Policy advocacy engagement 17 organizational factors 46 559 Not applicable Organizational size, professionalization, board support, constituent involvement, knowledge about laws, government funding, private donations, foundation funding, collaboration, and negative policy environment each has a positive association with a nonprofit’s level of advocacy engagement
Lu and Xu (2018) Voluntas Nonprofit sector size Government size 30 151 Nonprofit sector size measure (density, finance, employment, etc.), government size measure (finance, employment, etc.), data structure (cross-sectional versus others), unit of analysis (country, state, county, city), country location (USA versus other nations), field of activity (social services versus others) The relationship between the two variables ranges from null to slight positive
Variable measurement and country location do not moderate the relationship
Data structure, unit of analysis, and field of activity moderate the relationship
Hung and Hager (2019) NVSQ Financial health Revenue diversification 40 296 Financial health measure (capacity versus others), revenue diversification measure (three revenue sources versus others), analysis type (bivariate versus multivariate), endogeneity correction (fixed-effect model versus others), publication era (published in/after 2011 versus others), country location (USA versus other nations), nonprofit subsector There is a small, positive association between the two variables
Revenue diversification measure, country location, publication era moderate the relationship
Lu et al. (2019) Voluntas Financial capacity and vulnerability Revenue diversification 23 258 Years under study, number of revenue sources, control for service area, control for size, control for age Revenue diversification has no association with financial vulnerability, but a small, negative association with financial capacity
Years under study, number of revenue sources, control for policy field, and control for size moderate the relationships
Hung (2020) NML Nonprofit donations Commercialization 25 295 Revenue types, subsector, use of local-level controls, locations, data collecting year, research design variables, and study characteristics Commercialization crowds out donations
Mission-driven commercial revenues return a more negative effect
International development and public benefit nonprofits return a more negative effect
Studies using longitudinal data demonstrate a more positive effect
Chapman et al. (2021) NVSQ Charitable Giving Trust 42 69 Type of trust, giving type, sample, region, publication status, year of data collection There is positive association between trust and giving
Organizational and sectoral trust were more strongly related to giving than were generalized or institutional trust
The relationship was also stronger in non-western countries and in non-representative samples
M. J., & Gillespie, N. (2021)       publication status, year of data collection Organizational and sectoral trust were more strongly related to giving than were generalized or institutional trust.
The relationship was also stronger in non-western countries and in nonrepresentative samples.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hung, C., Lu, J. Meta-Analysis for Nonprofit Research: Synthesizing Quantitative Evidence for Knowledge Advancement. Voluntas (2022). https://doi.org/10.1007/s11266-022-00505-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11266-022-00505-3

Keywords

  • Meta-analysis
  • Research synthesis
  • Nonprofit studies
  • Quantitative research methods