Skip to main content
SpringerLink
Log in

Meta-Analysis for Nonprofit Research: Synthesizing Quantitative Evidence for Knowledge Advancement

  • Research Papers
  • Published:
VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations Aims and scope Submit manuscript

Abstract

The past two decades have witnessed massive growth in the amount of quantitative research in nonprofit studies. Despite the large number of studies, findings from these studies have not always been consistent and cumulative. The diverse and competing findings constitute a barrier to offering clear, coherent knowledge for both research and practice. To further advance nonprofit studies, some have called for meta-analysis to synthesize inconsistent findings. Although meta-analysis has been increasingly used in nonprofit studies in the past decade, many researchers are still not familiar with the method. This article thus introduces meta-analysis to nonprofit scholars and, through an example demonstration, provides general guidelines for nonprofit scholars with background in statistical methods to conduct meta-analyses, with a focus on various judgement calls throughout the research process. This article could help nonprofit scholars who are interested in using meta-analysis to address some unsolved research questions in the nonprofit literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. In this manuscript, we use nonprofit studies to refer to the studies on voluntary actions, nonprofit organizations, and civil society.

  2. We also searched some other related journals in nonprofit studies including International Review on Public and Nonprofit Marketing, Journal of Nonprofit & Public Sector Marketing, Journal of Philanthropy and Marketing, Journal of Nonprofit Education and Leadership, Nonprofit Policy Forum, and only found one meta-analysis (Xu & Huang, 2020).

  3. This article introduces the basic steps and important judgment calls for nonprofit scholars who are interested in conducting meta-analysis. The discussion in this section is not exhaustive. Interested readers should refer to meta-analysis textbooks listed in the references.

  4. Ringquist (2013, pp. 121 to 124) provides detailed information about choosing appropriate statistics to estimate effect sizes.

  5. Before analyzing effect sizes, researchers need to choose between a fixed effects and a random effects framework. There are two approaches that researchers can use to decide which framework to use. They are the Q test and I2 statistic approaches. Researchers can use the Q test approach to identify excess variance in a sample of effect sizes, and use the I2 statistic approach to assess the magnitude of the variability in effect sizes that is not attributable to sampling errors. Ringquist (2013, pp. 121 to 124) offers detailed information to conduct the Q test and calculate I2 statistic, providing criteria to choose between the two frameworks. Generally, random effects models are more widely used in social science, since it is less reasonable to assume that all studies shares the same, one common effect.

  6. The inverse variance weight is a function of sample size. Effect sizes from studies that have larger sample are placed more weight.

  7. There might be effects that lie an abnormal distance from other effects. They are outliers. We suggest that researchers exclude them before combing effect size or conduct robustness checks by reporting average effect sizes with and without those outliers.

References

  • Blom, R., Kruyen, P. M., Van der Heijden, B. I., & Van Thiel, S. (2020). One HRM fits all? A meta-analysis of the effects of HRM practices in the public, semipublic, and private sector. Review of Public Personnel Administration, 40(1), 3–35.

    Article  Google Scholar 

  • Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–236). Russell Sage.

    Google Scholar 

  • Cheung, M. W. L. (2015). Meta-analysis: A structural equation modeling approach. John Wiley & Sons.

    Book  Google Scholar 

  • Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12–37.

    Article  Google Scholar 

  • Chapman, C. M., Hornsey, M. J., & Gillespie, N. (2021). To what extent is trust a prerequisite for charitable giving? A systematic review and meta-analysis. Nonprofit and Voluntary Sector Quarterly, 50, 1274–1303. https://doi.org/10.1177/08997640211003250

    Article  Google Scholar 

  • Cooper, H. (2017). Research synthesis and meta-analysis: A step-by-step approach. Sage.

    Book  Google Scholar 

  • Daniel, J. L., & Kim, M. (2018). The scale of mission-embeddedness as a nonprofit revenue classification tool: Different earned revenue types, different performance effects. Administration & Society, 50(7), 947–972.

    Article  Google Scholar 

  • De Wit, A., & Bekkers, R. (2017). Government support and charitable donations: A meta-analysis of the crowding-out hypothesis. Journal of Public Administration Research and Theory, 27(2), 301–319.

    Google Scholar 

  • Fleiss, J. L., & Berlin, J. A. (2009). Effect sizes for dichotomous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 237–254). Russell Sage.

    Google Scholar 

  • Geyskens, I., Krishnan, R., Steenkamp, J. B. E., & Cunha, P. V. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35(2), 393–419.

    Article  Google Scholar 

  • Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8.

    Article  Google Scholar 

  • Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018). Meta-analysis and the science of research synthesis. Nature, 555(7695), 175–182.

    Article  Google Scholar 

  • Hedges, L., & Olkin, I. (1985). Statistical methods for meta-analysis. Academic Press.

    Google Scholar 

  • Hung, C. (2020). Commercialization and nonprofit donations: A meta-analytic assessment and extension. Nonprofit Management and Leadership. https://doi.org/10.1002/nml.21435

    Article  Google Scholar 

  • Hung, C., & Hager, M. A. (2019). The impact of revenue diversification on nonprofit financial health: A meta-analysis. Nonprofit and Voluntary Sector Quarterly, 48(1), 5–27.

    Article  Google Scholar 

  • Hunt, M. (1997). How science takes stock: The story of meta-analysis. Russell Sage Foundation.

    Google Scholar 

  • Jackson, S. K., Guerrero, S., & Appe, S. (2014). The state of nonprofit and philanthropic studies doctoral education. Nonprofit and Voluntary Sector Quarterly, 43(5), 795–811.

    Article  Google Scholar 

  • Kim, M. (2017). The relationship of nonprofits’ financial health to program outcomes: Empirical evidence from nonprofit arts organizations. Nonprofit and Voluntary Sector Quarterly, 46(3), 525–548.

    Article  Google Scholar 

  • Kulik, J. A., & Kulik, C. L. C. (1989). Meta-analysis in education. International Journal of Educational Research, 13(3), 221–340.

    Article  Google Scholar 

  • Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Sage.

    Google Scholar 

  • Lu, J. (2016). The philanthropic consequence of government grants to nonprofit organizations: A meta-analysis. Nonprofit Management and Leadership, 26(4), 381–400.

    Article  Google Scholar 

  • Lu, J. (2017). Does population heterogeneity really matter to nonprofit sector size? Revisiting Weisbrod’s demand heterogeneity hypothesis. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 1–27.

  • Lu, J. (2018). Organizational antecedents of nonprofit engagement in policy advocacy: A meta-analytical review. Nonprofit and Voluntary Sector Quarterly, 47(4_suppl), 177S-203S.

    Article  Google Scholar 

  • Lu, J., & Xu, C. (2018). Complementary or supplementary? The relationship between government size and nonprofit sector size. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 29(3), 454–469.

    Article  Google Scholar 

  • Lu, J., Lin, W., & Wang, Q. (2019). Does a more diversified revenue structure lead to greater financial capacity and less vulnerability in nonprofit organizations? A bibliometric and meta-analysis. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 30(3), 593–609.

    Article  Google Scholar 

  • Ma, J., & Konrath, S. (2018). A century of nonprofit studies: Scaling the knowledge of the field. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 29(6), 1139–1158.

    Article  Google Scholar 

  • Pearson, K. (1904). Report on certain enteric fever inoculation statistics. BMJ, 3, 1243–1246.

    Google Scholar 

  • Pfeffer, J. (1993). Barriers to the advance of organizational science: Paradigm development as a dependent variable. Academy of Management Review, 18(4), 599–620.

    Article  Google Scholar 

  • Reed, J. G., & Baxter, P. M. (2009). Using reference databases. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 73–101). Russell Sage Foundation.

    Google Scholar 

  • Ringquist, E. (2013). Meta-analysis for public management and policy. John Wiley & Sons.

    Google Scholar 

  • Rosethal, R., & DiMatteo, M. (2001). Meta-analysis: Recent developments in quantitative methods for literature review. Annual Review of Psychology, 52, 59–82.

    Article  Google Scholar 

  • Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2006). Publication bias in meta-analysis: Prevention, assessment and adjustments. John Wiley & Sons.

    Google Scholar 

  • Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173–1181.

    Article  Google Scholar 

  • Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings. Sage.

    Book  Google Scholar 

  • Shadish, W. R., & Lecy, J. D. (2015). The meta-analytic big bang. Research Synthesis Methods, 6(3), 246–264.

    Article  Google Scholar 

  • Shoham, A., Ruvio, A., Vigoda-Gadot, E., & Schwabsky, N. (2006). Market orientations in the nonprofit and voluntary sector: A meta-analysis of their relationships with organizational performance. Nonprofit and Voluntary Sector Quarterly, 35(3), 453–476.

    Article  Google Scholar 

  • Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752–760.

    Article  Google Scholar 

  • Stanley, T. D., & Jarrell, S. B. (2005). Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 19(3), 299–308.

    Article  Google Scholar 

  • Sutton, A. J. (2009). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435–452). Russell Sage Foundation.

    Google Scholar 

  • Sutton, A. J., Abrams, K. R., Jones, D. R., Jones, D. R., Sheldon, T. A., & Song, F. (2000). Methods for meta-analysis in medical research. Wiley.

    Google Scholar 

  • Thompson, S. G., & Higgins, J. P. (2002). How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21(11), 1559–1573.

    Article  Google Scholar 

  • Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10(2), 161–179.

    Article  Google Scholar 

  • Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222.

    Article  Google Scholar 

  • Xu, J., & Huang, G. (2020). The relative effectiveness of gain-framed and loss-framed messages in charity advertising: Meta-analytic evidence and implications. International Journal of Nonprofit and Voluntary Sector Marketing, 25(4), e1675.

    Article  Google Scholar 

  • Willems, J., Boenigk, S., & Jegers, M. (2014). Seven trade-offs in measuring nonprofit performance and effectiveness. Voluntas: International Journal of Voluntary and Nonprofit Organizations, 25(6), 1648–1670.

    Article  Google Scholar 

Download references

Funding

This study was not funded by any organization.

Author information

Authors and Affiliations

  1. Public Administration Program, University of Hawaiʻi at Mānoa, 2424 Maile Way, Saunders Hall 631, Honolulu, HI, 96822, USA

    ChiaKo Hung

  2. School of Public Affairs and Administration, Rutgers University-Newark, 111 Washington Street, Newark, NJ, 07102, USA

    Jiahuan Lu

Authors
  1. ChiaKo Hung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiahuan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to ChiaKo Hung.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interests.

Human and Animal Rights

This study does not involve Human Participants and/or Animals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix 1. Published Meta-Analyses in Three Leading Nonprofit Studies Journals

Appendix 1. Published Meta-Analyses in Three Leading Nonprofit Studies Journals

Study

Journal

Dependent variable

Focal predictors

# studies included

# effect sizes analyzed

Moderators

Main findings

Shoham et al. (2006)

NVSQ

Organizational performance

Market orientation

11

11

Country location (USA versus other nations), market orientation measure (behavioral versus philosophical measure), performance measure (subjective versus objective measures)

There is a positive association between the two variables

The relationship is stronger in nonprofits than for-profits

Country location and market orientation measure moderate the relationship

Lu (2016)

NML

Private donations

Government grants

60

637

Nonprofit subsector, country location (USA versus other nations), control for age, control for size, data structure (longitudinal versus others), lagged effect, endogeneity correction

There is a small, positive correlation between the two variables

Nonprofit subsector, control for age, data structure, and endogeneity moderate the relationship

Lu (2017)

Voluntas

Nonprofit sector size

Population heterogeneity

37

491

Country location (USA versus other nations, within-country versus cross-country), nonprofit sector size measure (density, finance, employment, etc.), population heterogeneity measure (age, employment status, ethnicity, gender, etc.)

There is a small, positive association between the two variables

The relationship is generalized across countries and measurements of nonprofit sector size

Population heterogeneity in terms of age, education, ethnicity, language, and religion predicts nonprofit sector size better

Lu (2018)

NVSQ

Policy advocacy engagement

17 organizational factors

46

559

Not applicable

Organizational size, professionalization, board support, constituent involvement, knowledge about laws, government funding, private donations, foundation funding, collaboration, and negative policy environment each has a positive association with a nonprofit’s level of advocacy engagement

Lu and Xu (2018)

Voluntas

Nonprofit sector size

Government size

30

151

Nonprofit sector size measure (density, finance, employment, etc.), government size measure (finance, employment, etc.), data structure (cross-sectional versus others), unit of analysis (country, state, county, city), country location (USA versus other nations), field of activity (social services versus others)

The relationship between the two variables ranges from null to slight positive

Variable measurement and country location do not moderate the relationship

Data structure, unit of analysis, and field of activity moderate the relationship

Hung and Hager (2019)

NVSQ

Financial health

Revenue diversification

40

296

Financial health measure (capacity versus others), revenue diversification measure (three revenue sources versus others), analysis type (bivariate versus multivariate), endogeneity correction (fixed-effect model versus others), publication era (published in/after 2011 versus others), country location (USA versus other nations), nonprofit subsector

There is a small, positive association between the two variables

Revenue diversification measure, country location, publication era moderate the relationship

Lu et al. (2019)

Voluntas

Financial capacity and vulnerability

Revenue diversification

23

258

Years under study, number of revenue sources, control for service area, control for size, control for age

Revenue diversification has no association with financial vulnerability, but a small, negative association with financial capacity

Years under study, number of revenue sources, control for policy field, and control for size moderate the relationships

Hung (2020)

NML

Nonprofit donations

Commercialization

25

295

Revenue types, subsector, use of local-level controls, locations, data collecting year, research design variables, and study characteristics

Commercialization crowds out donations

Mission-driven commercial revenues return a more negative effect

International development and public benefit nonprofits return a more negative effect

Studies using longitudinal data demonstrate a more positive effect

Chapman et al. (2021)

NVSQ

Charitable Giving

Trust

42

69

Type of trust, giving type, sample, region, publication status, year of data collection

There is positive association between trust and giving

Organizational and sectoral trust were more strongly related to giving than were generalized or institutional trust

The relationship was also stronger in non-western countries and in non-representative samples

M. J., & Gillespie, N. (2021)

     

publication status, year of data collection

Organizational and sectoral trust were more strongly related to giving than were generalized or institutional trust.

The relationship was also stronger in non-western countries and in nonrepresentative samples.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hung, C., Lu, J. Meta-Analysis for Nonprofit Research: Synthesizing Quantitative Evidence for Knowledge Advancement. Voluntas 34, 734–746 (2023). https://doi.org/10.1007/s11266-022-00505-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11266-022-00505-3

Keywords

Search

Navigation