Skip to main content

Advertisement

SpringerLink
Log in

Feasibility study and evaluation of expert opinion on the semi-automated meta-analysis and the conventional meta-analysis

  • Pharmacoepidemiology and Prescription
  • Published:
European Journal of Clinical Pharmacology Aims and scope Submit manuscript

Abstract

Purpose

To assess the feasibility and acceptance of the semi-automated meta-analysis (SAMA). The objectives are twofold, namely (1) to compare expert opinion on the quality of protocols, methods, and results of one conventional meta-analysis (CMA) and one SAMA and (2) to compare the time to execute the CMA and the SAMA.

Methods

Experts evaluated the protocols and manuscripts/reports of the CMA and SAMA conducted independently on the safety of metronidazole in pregnancy. Expert opinion was collected using AMSTAR 2 checklist. Time spent was recorded using case report forms.

Results

The overall scores of the opinion of all experts for protocols, methods, and results for SAMA (6.75) and CMA (6.87) were not statistically different (p = 0.88). The experts’ confidence in the results of each MA was 7.89 ± 1.17 and 8.11 ± 0.92, respectively. The time to completion was 14 working days for SAMA and 24.7 for CMA. MA tasks such as calculation of effect estimates, subgroup/sensitivity analysis, and publication bias investigation required no investment in time for SAMA.

Conclusion

In conclusion, our study demonstrated the feasibility of SAMA and suggests acceptance for risk assessment by an expert committee. Our results suggest that SAMA reduces the time required for a MA without altering expert confidence in the methodological and scientific rigor. As our study was limited to one example, the generalization of our results requires confirmation by other studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Haidich AB (2010) Meta-analysis in medical research. Hippokratia 14(Suppl 1):29–37

    CAS  PubMed  PubMed Central  Google Scholar 

  2. Greco T, Zangrillo A, Biondi-Zoccai G, Landoni G (2013) Meta-analysis: pitfalls and hints. Heart Lung Vessels 5(4):219–225

    CAS  PubMed  PubMed Central  Google Scholar 

  3. Bastian H, Glasziou P, Chalmers I (2010) Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 7(9):e1000326

    Article  Google Scholar 

  4. Allen IE, Olkin I (1999) Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA 282(7):634–635

    Article  CAS  Google Scholar 

  5. Borah R, Brown AW, Capers PL, Kaiser KA (2017) Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 7(2):e012545

    Article  Google Scholar 

  6. Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S et al (2017) Living systematic review: 1. Introduction—the why, what, when, and how. J Clin Epidemiol 91:23–30

    Article  Google Scholar 

  7. Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C et al (2017) Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol 91:31–37

    Article  Google Scholar 

  8. Safi S, Thiessen T, Schmailzl KJ (2018) Acceptance and resistance of new digital technologies in medicine: qualitative study. JMIR Res Protoc 7(12):e11072

    Article  Google Scholar 

  9. O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B (2019) A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev 8:143

    Article  Google Scholar 

  10. Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM (2020) A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol 121:81–90

    Article  Google Scholar 

  11. O’Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Wolfe MS (2018) Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev 7(1):3

    Article  Google Scholar 

  12. Marshall IJ, Wallace BC (2019) Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev 8(1):163

    Article  Google Scholar 

  13. Gates A, Gates M, DaRosa D, Elliott SA, Pillay J, Rahman S et al (2020) Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews. Syst Rev 9(1):272

    Article  Google Scholar 

  14. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J et al (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008. [Accessed 13 févr 2020];358.  https://www.bmj.com/content/358/bmj.j4008

  15. Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A et al (2017) Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study. Syst Rev 6(1):117

    Article  Google Scholar 

  16. Hemming V, Burgman MA, Hanea AM, McBride MF, Wintle BC (2018) A practical guide to structured expert elicitation using the IDEA protocol. Methods Ecol Evol 9(1):169–180

    Article  Google Scholar 

  17. Speirs-Bridge A, Fidler F, McBride M, Flander L, Cumming G, Burgman M (2010) Reducing overconfidence in the interval judgments of experts. Risk Anal Off Publ Soc Risk Anal 30(3):512–523

    Google Scholar 

  18. R Core Team (2020) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria [Internet]. 2020 [Accessed 30 mars 2020] https://www.r-project.org/

  19. Viechtbauer W (2010) Conducting meta-analyses in R with the metafor package. J Stat Softw 36(1):1–48

    Google Scholar 

  20. Balduzzi S, Rücker G, Schwarzer G (2019) How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health 22(4):153–160

    Article  Google Scholar 

  21. Collignon A, Cuxac P (2017) ISTEX: des enrichissements au web de données. I2D - Inf Donnees Doc 54(4):8‑15

  22. Higgins JP, Green S (2008) Cochrane Handbook for Systematic Reviews of Interventions: Chichester, UK: John Wiley & Sons, Ltd. http://doi.wiley.com/10.1002/9780470712184

  23. van Altena AJ, Spijker R, Olabarriaga SD (2019) Usage of automation tools in systematic reviews. Res Synth Methods 10(1):72–82

    Article  Google Scholar 

  24. Chai KEK, Lines RLJ, Gucciardi DF et al (2021) Research screener: a machine learning tool to semi-automate abstract screening for systematic reviews | SpringerLink [Internet]. [Accessed 28 avr 2021]. https://link.springer.com/article/10.1186/s13643-021-01635-3

  25. Low J, Ross JS, Ritchie JD, Gross CP, Lehman R, Lin H et al (2017) Comparison of two independent systematic reviews of trials of recombinant human bone morphogenetic protein-2 (rhBMP-2): the Yale Open Data Access Medtronic Project. Syst Rev 6(1):28

    Article  Google Scholar 

  26. Wiseman MJ (2008) Reproducibility of systematic literature reviews on food, nutrition, physical activity and endometrial cancer. Public Health Nutrition, 11 (10). 1006 - 1014 . ISSN 1368-9800 https://doi.org/10.1017/S1368980007001334 [Accessed 24 févr 2021] https://core.ac.uk/reader/13505414?utm_source=linkout

Download references

Acknowledgements

We thank the following experts of the ANSM Scientific Expert Committee for their contribution to this project:

Dr Florence Gressier MD PhD HDR: CESP, Inserm UMR1178, Department of Psychiatry, Assistance Publique-Hôpitaux de Paris, Bicêtre University Hospital, Le Kremlin Bicêtre, France. Dr Isabelle Lacroix: REGARDS Network, Medical and Clinical Pharmacology Department, Midi-Pyrenees Regional Centre for Pharmacovigilance, Pharmacoepidemiology and Drug Information (CRPV), Toulouse University Hospital, University of Toulouse, Inserm 1027. Dr Kim an Nguyen: Department of Pharmacotoxicology, Clinical Investigation Centre 1407, Inserm, Hospices Civils de Lyon, 69003 Lyon, France; UMR5558 CNRS, Laboratoire de Biométrie et Biologie Evolutive, Université de Lyon, 69008 Lyon, France. Dr Nadine Saleh PharmD MS PhD: Head, Master of Public Health, Faculty of Public Health, Lebanese University, Lebanon. Dr Sophie Gautier: Centre Regional de Pharmacovigilance, Centre Hospitalier Universitaire de Lille, Lille, France. Dr Thierry Vial MD: Service Hospitalo-Universitaire de Pharmacotoxicologie, CHU-Lyon, Lyon, France.

Author information

Authors and Affiliations

  1. Agence Nationale de Sécurité du Médicament et des Produits de Santé (ANSM), 143 Boulevard Anatole France, 93200, Saint Denis, France

    Priscilla Ajiji, Anil Uzunali, Emmanuelle Ripoche & Patrick Maison

  2. EA 7379, Faculté de Santé, Université Paris-Est Créteil, Créteil, France

    Priscilla Ajiji & Patrick Maison

  3. Service Hospitalo-Universitaire de Pharmacotoxicologie, Hospices Civils de Lyon, Lyon, France

    Judith Cottin, Cyndie Picot & Michel Cucherat

  4. Laboratoire de Biométrie et Biologie Evolutive, UMR5558, CNRS, Université Lyon 1, 69008, Lyon, France

    Michel Cucherat

  5. CHI Créteil, Créteil, France

    Patrick Maison

Authors
  1. Priscilla Ajiji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Judith Cottin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cyndie Picot
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anil Uzunali
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emmanuelle Ripoche
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michel Cucherat
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Patrick Maison
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

PA: conceptualization, methodology, validation, visualization, formal analysis, data curation, writing — original draft preparation. JC: conceptualization, data curation, validation, writing — reviewing and editing. CP: data curation, validation, reviewing and editing. AU: data curation, validation. ER: conceptualization, validation. MC: conceptualization, software. PM: conceptualization, methodology, validation, writing — reviewing and editing, supervision.

Corresponding author

Correspondence to Patrick Maison.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 836 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ajiji, P., Cottin, J., Picot, C. et al. Feasibility study and evaluation of expert opinion on the semi-automated meta-analysis and the conventional meta-analysis. Eur J Clin Pharmacol 78, 1177–1184 (2022). https://doi.org/10.1007/s00228-022-03329-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00228-022-03329-8

Keywords

Search

Navigation