Meta-analysis of survey data: application to health services research

  • Sowmya R. RaoEmail author
  • Barry I. Graubard
  • Christopher H. Schmid
  • Sally C. Morton
  • Thomas A. Louis
  • Alan M. Zaslavsky
  • Dianne M. Finkelstein


Traditionally, meta-analysis methods have been developed and used to combine data from several independent clinical trials as well as observational studies, but have not been as widely used in survey research. This paper describes the steps in conducting such a meta-analysis of surveys, to obtain a single summary estimate from a combination of individual-level and summary data. The methods are applied in the context of a project aimed at obtaining an estimate of the prevalence of use of electronic health records.


Meta-analysis Surveys Health research Electronic health record 



The authors thank Dr. David Blumenthal, Dr. Karen Donelan, Dr. Catherine DesRoches, and the HIT Adoption Initiative team for their help. We also thank Catherine W. Burt and Esther S. Hing from NCHS for providing us with the estimates for the group practices using the 2005 NAMCS data. S.R.R. and D.M.F. were supported by a contract from the Office of the National Coordinator for Health Information Technology and the Robert Wood Johnson Foundation. Funding: The project was supported by the Office of the National Coordinator for Health Information Technology, Washington, D.C. and the Robert Wood Johnson Foundation, Princeton, N.J.


  1. AAPOR. Code of Professional Ethics and Practices (2005)Google Scholar
  2. American Association for Public Opinion Research.: Best Practices for Survey and Public Opinion Research (1997)Google Scholar
  3. Armitage, P.: Inference and decision in clinical trials. J. Clin. Epidemiol. 42, 293–299 (1989)PubMedCrossRefGoogle Scholar
  4. Audet, A.M., Doty, M., Peugh, J., Shamasdin, J., Zapert, K., Schoenbaum, S.: Information technologies: When will they make it into physicians’ black bags? Med. Gen. Med. 6(4), 2 (2004)Google Scholar
  5. Berlin, J.A., Rennie, D.: Measuring the quality of trials: the quality of quality scales. JAMA 282(11), 1083–1085 (1999)PubMedCrossRefGoogle Scholar
  6. Blumenthal, D., DesRoches, C., Donelan, K., Ferris, T., Jha, A., Kaushal, R., Rao, S., Rosenbaum S., Shield, A.: Health information technology in the United States: The information base for progress. Robert Wood Foundation (2006)Google Scholar
  7. Burt, C.W., Hing, E., Woodwell, D.: Electronic medical record use by office-based physicians: United States, 2005. National Center for Health Statistics, Hyattsville, MD. Available at: Accessed April 16, 2008
  8. Buuren, S.V., Eyres, S., Tennant, A., Hopman-Rock, M.: Improving comparability of existing data by response conversion. JOS 21, 53–72 (2005)Google Scholar
  9. Chalmers, T.C.: Problems induced by meta-analyses. Stat. Med. 10, 971–980 (1991)PubMedCrossRefGoogle Scholar
  10. Cochran, W.G.: Sampling Techniques. John Wiley & Sons, New York, NY (1977)Google Scholar
  11. Cooper, H., Hedges, L. (eds.): The Handbook of Research Synthesis. Russell Sage Foundation (1994)Google Scholar
  12. DerSimonian, R., Laird, N.: Meta-analysis in clinical trials. Control. Clin. Trials 7, 177–188 (1986)PubMedCrossRefGoogle Scholar
  13. Devereaux, P.J., Choi, P.T., Lacchetti, C., Weaver, B., Schünemann, H.J., Haines, T., Lavis, J.N., Grant, B.J., Haslam, D.R., Bhandari, M., Sullivan, T., Cook, D.J., Walter, S.D., Meade, M., Khan, H., Bhatnagar, N., Guyatt, G.H.: A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. CMAJ 166(11), 1399–1406 (2002)Google Scholar
  14. Egger, M., Smith, G.D.: Meta-analysis bias in location and selection of studies. BMJ 316, 61–66 (1998)PubMedGoogle Scholar
  15. Elliott, M.R., Davis, W.W.: Obtaining cancer risk factor prevalence estimates in small areas: combining data from two surveys. JRSS Appl. Stat. 54(3), 595–609 (2005)Google Scholar
  16. Fuller, W.A, Burmeister, L.F.: Estimators for samples selected from two overlapping frames. Proceedings of the Social Statistics Section, American Statistical Association, 245–249 (1972)Google Scholar
  17. Gans, D.N., Kralewski, J.E., Hammons, T., Dowd, B.: Medical groups’ adoption of electronic health records and information systems. Health Aff. 24, 1323–1333 (2005)CrossRefGoogle Scholar
  18. Graham, D.: Analysis of Variance. In: Armitage, P., Colton, T. (eds.) Encyclopedia of Biostatistics, vol. 1, pp. 2574–2576. Wiley, New York (1998)Google Scholar
  19. Groves, R.M., Dillman, D.A., Eltinge, J.L., Little, R.J.A. (eds.): Survey Nonresponse. John Wiley & Sons, New York (2002)Google Scholar
  20. Hartley, H.O.: Multiple frame surveys. Proceedings of the Social Statistics Section, American Statistical Association, 203–206 (1962)Google Scholar
  21. Hartley, H.O.: Multiple frame methodology and selected applications. Sankhya C 36, 99–118 (1974)Google Scholar
  22. Higgins, J.P.T., Thompson, S.G.: Quantifying heterogeneity in a meta-analysis. Stat. Med. 21, 1539–1558 (2002)PubMedCrossRefGoogle Scholar
  23. Higgins, J.P.T., Thompson, S.G., Deeks, J.J., Altman, D.G.: Measuring inconsistency in meta-analysis. BMJ 327, 557–560 (2003)PubMedCrossRefGoogle Scholar
  24. Jha, A.J, Ferris, T.G., Donelan, K., Desroches, C., Shields, A., Rosenbaum, S., Blumenthal, D.: How common are electronic health records in the United States? A summary of the evidence. Health Affairs Web Exclusive, October 11 (2006)Google Scholar
  25. Jüni, P., Witschi, A., Bloch, R., Egger M.: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282, 1054–1060 (1999)PubMedCrossRefGoogle Scholar
  26. Kadane, J.: Some statistical problems in merging data files. Compendium of Tax Research, U.S. Department of the Treasury, 159–171 (1978) (Reprinted in JOS 17(3), 423–433 (2001))Google Scholar
  27. Korn, E.L., Graubard, B.I.: Analysis of Health Surveys. John Wiley & Sons, New York, NY (1999)Google Scholar
  28. Lambert, P.C., Sutton, A.J., Abrams, K.R., Jones, D.R.: A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J. Clin. Epidemiol. 55, 86–94 (2002)PubMedCrossRefGoogle Scholar
  29. Lan, G.K.K., Hu, M., Cappelleri, J.C.: Applying the law of iterated logarithm to cumulative meta-analysis of a continuous endpoint. Stat. Sin. 13, 1135–1145 (2003)Google Scholar
  30. Lau, J., Antman, E.M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., Chalmers, T.C.: Cumulative meta-analysis of therapeutic trials for myocardial infarction. NEJM 327, 248–254 (1992)PubMedGoogle Scholar
  31. Lau, J., Schmid, C.H., Chalmers, T.C.: Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J. Clin. Epidemiol. 48, 45–57 (1995)PubMedCrossRefGoogle Scholar
  32. Lohr, S.L., Rao, J.N.K.: Inference from dual frame surveys. JASA 99, 271–280 (2000)Google Scholar
  33. Loomis, G.A., Ries, J.S., Saywell, R.M. Jr., Thakker, N.R.: If electronic medical records are so great, why aren’t family physicians using them? J. Fam. Pract. 51(7), 636–641 (2002)PubMedGoogle Scholar
  34. Lund, R.E.: Estimators in multiple frame surveys. Proceedings of the Social Statistics Section, American Statistical Association, 282–288 (1968)Google Scholar
  35. Moriarty, C., Scheuren, F.: Statistical matching: a paradigm for assessing the uncertainty in the procedure. JOS 17(3), 407–422 (2001)Google Scholar
  36. Rodgers, W.L.: An evaluation of statistical matching. J. Bus. Econ. Stat. 2, 91–102 (1984)CrossRefGoogle Scholar
  37. Rubin, D.B.: Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons, New York (1987)Google Scholar
  38. Schenker, N., Gentleman, J.F., Rose, D., Hing, E., Schimizu, I.: Combining estimates from complementary surveys: a case study using prevalence estimates from national health surveys of households and nursing homes. Public Health Rep. 117, 393–407 (2002)PubMedGoogle Scholar
  39. Schenker, N., Raghunathan, T.E.: Combining information from multiple surveys to enhance estimation of measures of health. Stat. Med. 26(8), 1802–1811 (2007)PubMedCrossRefGoogle Scholar
  40. Scheuren, F.: What is a survey? Accessed April 16, 2008
  41. Schmid, C.H., Landa, M., Jafar, T.H., Giatras, I., Karim, T., Reddy, M., Stark, P.C., Levey, A.S. for the Angiotensin-Converting Enzyme Inhibition in Progressive Renal Disease (AIPRD) Study Group.: Constructing a database of individual clinical trials for longitudinal analysis. Control. Clin. Trials 24, 324–340 (2003)Google Scholar
  42. Schmid, C.H., Stark, P.C., Berlin, J.A., Landais, P., Lau, J.: Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. J. Clin. Epidemiol. 57, 683–697 (2004)PubMedCrossRefGoogle Scholar
  43. Stroup, D.F., Berlin, J.A., Morton, S.C., Olkin, I., Williamson, G.D., Rennie, D., Moher, D., Becker, B.J., Sipe, T.A., Thacker, S.T. for the Meta-analysis of Observational Studies in Epidemiology (MOOSE) Group.: Meta-analysis of observation studies in epidemiology: a proposal for reporting. JAMA 283(15), 2008–2012 (2000)Google Scholar
  44. Terry, K.: Exclusive survey: Doctors and EHRs. Medical Economics (2005). Available at: Accessed April 16, 2008
  45. U.S. Dept. of Commerce, Bureau of the Census. Census Quality Survey Public Use Data File, 2001: [United States] [Computer file]. ICPSR release. Washington, DC: U.S. Dept. of Commerce, Bureau of the Census [producer], 2003. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], (2004)Google Scholar
  46. Willis, G.B.: Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage Publications, Thousand Oaks, CA (2005)Google Scholar
  47. Zaslavsky, A.M., Shaul, J.A., Zaborski, L.B., Cioffi, M.J., Cleary, P.D.: Combining health plan performance indicators into simpler composite measures. Health Care Financ. Rev. 23(4), 101–115 (2002)PubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Sowmya R. Rao
    • 1
    Email author
  • Barry I. Graubard
    • 2
  • Christopher H. Schmid
    • 3
  • Sally C. Morton
    • 4
  • Thomas A. Louis
    • 5
  • Alan M. Zaslavsky
    • 6
  • Dianne M. Finkelstein
    • 7
  1. 1.MGH Biostatistics Center and The Institute for Health Policy, MGHBostonUSA
  2. 2.National Cancer InstituteBethesdaUSA
  3. 3.Tufts Sackler School of Graduate Biomedical SciencesBostonUSA
  4. 4.RTI InternationalResearch Triangle ParkUSA
  5. 5.Johns Hopkins Bloomberg School of Public HealthBaltimoreUSA
  6. 6.Department of Health Care PolicyHarvard Medical SchoolBostonUSA
  7. 7.MGH Biostatistics Center, MGHBostonUSA

Personalised recommendations