, Volume 8, Issue 4, pp 261–271 | Cite as

Achieving High Research Reporting Quality Through the Use of Computational Ontologies

  • Amrapali Zaveri
  • Luciana Cofiel
  • Jatin Shah
  • Shreyasee Pradhan
  • Edwin Chan
  • Olivier Dameron
  • Ricardo Pietrobon
  • Beng Ti Ang


Systematic reviews and meta-analyses constitute one of the central pillars of evidence-based medicine. However, clinical trials are poorly reported which delays meta-analyses and consequently the translation of clinical research findings to clinical practice. We propose a Center of Excellence in Research Reporting in Neurosurgery (CERR-N) and the creation of a clinically significant computational ontology to encode Randomized Controlled Trials (RCT) studies in neurosurgery. A 128 element strong computational ontology was derived from the Trial Bank ontology by omitting classes which were not required to perform meta-analysis. Three researchers from our team tagged five randomly selected RCT’s each, published in the last 5 years (2004–2008), in the Journal of Neurosurgery (JoN), Neurosurgery Journal (NJ) and Journal of Neurotrauma (JoNT). We evaluated inter and intra observer reliability for the ontology using percent agreement and kappa coefficient. The inter-observer agreement was 76.4%, 75.97% and 74.9% and intra-observer agreement was 89.8%, 80.8% and 86.56% for JoN, NJ and JoNT respectively. The inter-observer kappa coefficient was 0.60, 0.54 and 0.53 and the intra-observer kappa coefficient was 0.79, 0.82 and 0.79 for JoN, NJ and JoNT journals respectively. The high degree of inter and intra-observer agreement confirms tagging consistency in sections of a given scientific manuscript. Standardizing reporting for neurosurgery articles can be reliably achieved through the integration of a computational ontology within the context of a CERR-N. This approach holds potential for the overall improvement in the quality of reporting of RCTs in neurosurgery, ultimately streamlining the translation of clinical research findings to improvement in patient care.


Systematic review Meta-analyses Evidence-based medicine Reporting RCT Neurosurgery Standardized Ontology Kappa coefficient 



We would like to thank the Research on Research group (, Duke University, USA for manuscript writing templates. (Phadtare et al. 2009). We would also like to thank FS from the Cochrane group who helped us modify the ontology, by giving us input as to which classes would be required for a meta-analysis.


  1. Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., et al. (2001). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Annals of Internal Medicine, 134(8), 663–694.PubMedGoogle Scholar
  2. Andersen, J., Larsen, J. K., Schultz, V., Nielsen, B. M., Kørner, A., Behnke, K., et al. (1989). The brief psychiatric rating scale. Dimension of schizophrenia—reliability and construct validity. Psychopathology, 22(2–3), 168–176.PubMedCrossRefGoogle Scholar
  3. Biesen, W. V., Verbeke, F., & Vanholder, R. (2008). An infallible recipe? A story of cinnamon, soufflé and meta-analysis. Nephrology, Dialysis, Transplantation, 23(9), 2729–2732.PubMedCrossRefGoogle Scholar
  4. Bossuyt, P. (2004). The quality of reporting in diagnostic test research: getting better, still not optimal. Clinical Chemistry, 50, 465–466.PubMedCrossRefGoogle Scholar
  5. Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., et al. (2004). Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Family Practice, 21, 4–10.PubMedCrossRefGoogle Scholar
  6. Buttram, S., Wisniewski, S., Jackson, E., Adelson, D., Feldman, K., Bayir, H., et al. (2007). Multiplex assessment of cytokine and chemokine levels in cerebrospinal fluid following severe pediatric traumatic brain injury: effects of moderate hypothermia. Journal of Neurotrauma, 24(11), 1707–1718. doi: 10.1089/neu.2007.0349.PubMedCrossRefGoogle Scholar
  7. Choi, G., Raiturker, P. P., Kim, M. J., Chung, D. J., Chae, Y. S., & Lee, S. H. (2005). The effect of early isolated lumbar extension exercise program for patients with herniated disc undergoing lumbar discectomy. Neurosurgery, 57, 764–772.PubMedCrossRefGoogle Scholar
  8. Clarke, M. J., & Stewart, L. A. (1994). Obtaining data from randomised controlled trials: how much do we need for reliable and informative meta-analyses? BMJ, 309(6960), 1007–1010.PubMedGoogle Scholar
  9. Cook, D. J., Greengold, N. L., Ellrodt, A. G., & Weingarten, S. R. (1997). The relation between systematic reviews and practice guidelines. Annals of Internal Medicine, 127(3), 210–216.PubMedGoogle Scholar
  10. Cook, C., Hannley, M., Richardson, J. K., Michon, J., Harker, M., & Pietrobon, R. (2007). Real-time updates of meta-analyses of HIV treatments supported by a biomedical ontology. Accountability in Research, 14(1), 1–18.PubMedCrossRefGoogle Scholar
  11. Crippa, J. A. S., Sanches, R. F., Hallak, J. E. C., Loureiro, S. R., & Zuardi, A. W. (2002). Factor structure of Bech’s version of the brief psychiatric rating scale in Brazilian patients. Brazilian Journal of Medical and Biological Research, 35(10), 1209–1213.PubMedCrossRefGoogle Scholar
  12. Fergusen, S., & Macdonald, R. L. (2007). Predictors of cerebral infarction in patients with aneurysmal subarachnoid hemorrhage. Neurosurgery, 60, 658–667.PubMedCrossRefGoogle Scholar
  13. Fonseca, F., & Martin, J. (2007). Learning the differences between ontologies and conceptual schemas through ontology-driven information systems. JAIS - Journal of Association for Information Systems - Special Issue on Ontologies in the Context of IS, 8(2), 129–142.Google Scholar
  14. Fung, A. E., Palanki, R., Bakri, S. J., Depperschmidt, E., & Gibson, A. (2009). Applying the CONSORT and STROBE statements to evaluate the reporting quality of neovascular age-related macular degeneration studies. Ophthalmology, 116(2), 286–296.PubMedCrossRefGoogle Scholar
  15. Groves, T. (2008). Enhancing the quality and transparency of health research. QBMJ, 337, a718.CrossRefGoogle Scholar
  16. Gruber, T. (2008). Ontology. Entry in the Encyclopedia of Database Systems. Ling Liu and M. Tamer Özsu (Eds.), Springer-Verlag, to appear in 2008. Provides a definition of ontology as a technical term for computer science, tracing its historical context from philosophy and AI.Google Scholar
  17. Gruber, T. R. (1993). Toward principles for the design of ontologies used for knowledge sharing.Google Scholar
  18. Haines, S. J. (1983). Randomized clinical trials in neurosurgery. Neurosurgery, 12(3), 259–264.PubMedCrossRefGoogle Scholar
  19. Han, C., Kwak, K. P., Marks, D. M., Pae, C. U., Wu, L. T., Bhatia, K. S., Masand, P. S., Patkar, A. A., et al. (2009). The impact of the CONSORT statement on reporting of randomized clinical trials in psychiatry. Contemporary Clinical Trials, 30(2), 116–122.PubMedCrossRefGoogle Scholar
  20. Hopewell, S., Altman, D. G., Moher, D., & Schulz, K. F. (2008). Endorsement of the CONSORT statement by high impact factor medical journals: a survey of journal editors and journal ‘Instructions to Authors’. Trials, 9, 20.PubMedCrossRefGoogle Scholar
  21. Hripcsak, G., & Heitjan, D. (2002). Measuring agreement in medical informatics reliability studies. Journal of Biomedical Informatics Archive, 35(2), 99–110.CrossRefGoogle Scholar
  22. Jarlais, D. C., Lyles, C., Crepaz, N., & TREND Group. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Public Health, 94, 361–366.CrossRefGoogle Scholar
  23. Jiang, J. Y., Wei, X. U., Li, W. P., Xu, W. H., Jun, Z., Bao, Y. H., et al. (2005). Efficacy of standard trauma craniectomy for refractory intracranial hypertension with severe traumatic brain injury: a multicenter, prospective, randomized controlled study. Journal of Neurotrauma, 22(6), 623–628.PubMedCrossRefGoogle Scholar
  24. Kane, R. L., Wang, J., & Garrard, J. (2007). Reporting in randomized clinical trials improved after adoption of the CONSORT statement. Journal of Clinical Epidemiology, 60(3), 241–249.PubMedCrossRefGoogle Scholar
  25. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.PubMedCrossRefGoogle Scholar
  26. Manchikanti, L., Derby, R., Wolfer, L., Singh, V., Datta, S., & Hirsch, J. A. (2008). Evidence based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Pain Physician, 11(2), 161–186.PubMedGoogle Scholar
  27. Manser, R., & Walters, E. H. (2001). What is evidence-based medicine and the role of the systematic review: the revolution coming your way. Monaldi Archives for Chest Diseases, 56(1), 33–38.Google Scholar
  28. Marmarou, A., Guy, M., Murphey, L., Roy, F., Layani, L., Combal, J. P., et al. (2005). A single dose, three-arm, placebo-controlled, phase I study of the bradykinin B2 receptor antagonist Anatibant (LF16-0687Ms) in patients with severe traumatic brain injury. Journal of Neurotrauma, 22(12), 1444–1455.PubMedCrossRefGoogle Scholar
  29. Mazzeo, A. T., Kunene, N. K., Gilman, C. B., Hamm, R. J., Hafez, N., & Bullock, M. R. (2006). Severe human traumatic brain injury, but not cyclosporin a treatment, depresses activated T lymphocytes early after injury. Journal of Neurotrauma, 23(6), 962–975.PubMedCrossRefGoogle Scholar
  30. Meinert, C. L. (1998). Beyond CONSORT: need for improved reporting standards for clinical trials. Consolidated standards of reporting trials. JAMA, 279(18), 1487–1489.PubMedCrossRefGoogle Scholar
  31. Meinert, C. L., Tonascia, S., & Higgins, K. (1984). Content of reports on clinical trials: a critical review. Controlled Clinical Trials, 5(4), 328–347.PubMedCrossRefGoogle Scholar
  32. Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., et al. (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet, 352(9128), 609–613.PubMedCrossRefGoogle Scholar
  33. Moher, D., Cook, D., Olkin, I., Stroup, D. F., & for the QUOROM Group. (2000). Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. QUOROM Group. The British Journal of Surgery, 87(11), 1448–1454.PubMedCrossRefGoogle Scholar
  34. Moher, D., Schulz, K. F., & Altman, D. G. (2001a). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet, 357(9263), 1191–1194.CrossRefGoogle Scholar
  35. Moher, D., Jones, A., Lepage, L., & CONSORT Group (Consolidated Standards for Reporting of Trials). (2001b). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA, 285(15), 1992.CrossRefGoogle Scholar
  36. Moher, D., Simera, I., Schulz, K. F., Hoey, J., & Altman, D. G. (2008). Helping editors, peer reviewers and authors improve the clarity, completeness and transparency of reporting health research. BMC Medicine, 6, 13.PubMedCrossRefGoogle Scholar
  37. Mummaneni, P. V., Burkus, J. K., Haid, R. W., Traynelis, V. C., & Zdeblick, T. A. (2007). Clinical and radiographic analysis of cervical disc arthroplasty compared with allograft fusion: a randomized controlled clinical trial. Neurosurgery Spine, 6, 198–209.CrossRefGoogle Scholar
  38. Naff, N. J., Hanley, D. F., Keyl, P. M., Tuhrim, S., Kraut, M., Bederson, J., et al. (2004). Intraventricular thrombolysis speeds blood clot resolution: results of a pilot, prospective, randomized, double-blind, controlled trial. Neurosurgery, 54, 577–584.PubMedCrossRefGoogle Scholar
  39. North, R. B., Kidd, D. H., Farrokhi, F., & Piantadosi, S. A. (2005). Spinal cord stimulation versus repeated lumbosacral spine surgery for chronic pain: a randomized, controlled trial. Neurosurgery, 56, 98–107.PubMedGoogle Scholar
  40. Noy, N. F., & McGuinness, D. L. (2001). Ontology Development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory Technical Report KSL-01-05 and Stanford Medical Informatics Technical Report. SMI-2001-0880.Google Scholar
  41. Petersson, H., Gill, H., & Ahlfeldt, H. (2002). A variance-based measure of inter-rater agreement in medical databases. Journal of Biomedical Informatics, 35(5–6), 331–342.PubMedCrossRefGoogle Scholar
  42. Phadtare, A., Bahmani, A., Shah, A., & Pietrobon, R. (2009). Scientific writing: a randomized controlled trial comparing standard and on-line instruction. BMC Medical Education, 9, 27.PubMedCrossRefGoogle Scholar
  43. Pocock, S. J., Hughes, M. D., & Lee, R. J. (1987). Statistical problems in the reporting of clinical trials. A survey of three medical journals. The New England Journal of Medicine, 317(7), 426–432.PubMedCrossRefGoogle Scholar
  44. Racer Pro 2.0 available at:
  45. Righesso, O., Falavigna, A., & Avanzi, O. (2007). Comparison of open discectomy with microendoscopic discectomy in lumbar disc herniations: results of a randomized controlled trial. Neurosurgery, 61, 545–549.PubMedCrossRefGoogle Scholar
  46. Rogers, J. E. (2006). Quality assurance of medical ontologies. Methods of Information in Medicine, 45(3), 267–274.PubMedGoogle Scholar
  47. Sackett, D. (1996). Evidence based medicine: what it is and what it isn’t. BMJ, 312, 71–72.PubMedGoogle Scholar
  48. Schulz, E., Barrett, J. W., & Colin, P. (1998). Read code quality assurance from simple syntax to semantic stability. Journal of the American Medical Informatics Association, 5, 337–346.PubMedGoogle Scholar
  49. Shankar, R. D., Martins, S. B., O’Connor, M., Parrish, D. B., & Das, A. K. (2007). An ontology-based architecture for integration of clinical trials management applications. AMIA Annu Symp Proc. 661–665.Google Scholar
  50. Sim, I., & Detmer, DE. (2005). Beyond trial registration: a global trial bank for clinical trial reporting. PLoS Medicine, 2(11), e365.PubMedCrossRefGoogle Scholar
  51. Sim, I., Owens, D. K., Lavori, P. W., & Rennels, G. D. (2000). Electronic trial banks: a complementary method for reporting randomized trials. Medical Decision Making, 20(4), 440–450.PubMedCrossRefGoogle Scholar
  52. Sim, I., Gorman, P., Greenes, R. A., Haynes, R. B., Kaplan, B., Lehmann, H., et al. (2001). Clinical decision support systems for the practice of evidence-based medicine. Journal of the American Medical Informatics Association, 8(6), 527–534.PubMedGoogle Scholar
  53. Sim, I., Carini, S., Olasov, B., & Jeng, S. (2004). An ontology of randomized controlled trials for evidence-based practice: content specification and evaluation using the competency decomposition method. Journal of Biomedical Informatics, 37(2), 108–119.PubMedCrossRefGoogle Scholar
  54. Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., et al. (2000). Meta-analysis of observational studies in epidemiology a proposal for reporting. JAMA, 283, 2008–2012.PubMedCrossRefGoogle Scholar
  55. Tapia-Perez, J. H., Sanchez-Aguilar, M., Torres-Corzo, J. G., Gordillo-Moscoso, A., Martinez-Perez, P., Madeville, P., et al. (2008). Effect of rosuvastatin on amnesia and disorientation after traumatic brain injury (NCT003229758). Journal of Neurotrauma, 25, 1011–1017.PubMedCrossRefGoogle Scholar
  56. Tendal, B., Higgins, J. P., Jüni, P., Hrobjartsson, A., Trelle, S., Nüesch, E. et al. (2008). Observer variation when extracting data for the calculation of a standardized mean difference, Poster presentation at 16th Cochrane Colloquium, Freiburg, 3-7 October 2008.Google Scholar
  57. Tseng, M. Y., Hutchinson, P. J., Turner, C. L., Czosnyka, M., Richards, H., Pickard, J. D., et al. (2007). Biological effects of acute pravastatin treatment in patients after aneurysmal subarachnoid hemorrhage: a double-blind, placebo-controlled trial. Journal of Neurosurgery, 107, 1092–1100.PubMedCrossRefGoogle Scholar
  58. Vajkoczy, P., Meyer, B., Weidauer, S., Raabe, A., Thome, C., Ringel, F., et al. (2005). Clazosentan (AXV-034343), a selective endothelin A receptor antagonist, in the prevention of cerebral vasospasm following severe aneurysmal subarachnoid hemorrhage: results of a randomized, double-blind, placebo-controlled, multicenter Phase IIa study. Journal of Neurosurgery, 103, 9–17.PubMedCrossRefGoogle Scholar
  59. Vranos, G., Tatsioni, A., Polyzoidis, K., Ioannidis, J. P. A., et al. (2004). Randomized trials of neurosurgical interventions: A systematic appraisal. Neurosurgery, 55(1), 18–26.PubMedCrossRefGoogle Scholar
  60. Wilczynski, N. L. (2008). Quality of reporting of diagnostic accuracy studies: no change since STARD statement publication—before-and-after study. Radiology, 248(3), 817–823.PubMedCrossRefGoogle Scholar
  61. Willems, P. W., Taphoorn, M. J., Burger, H., Berkelbach van der Sprenkel, J. W., & Tulleken, C. A. (2006). Effectiveness of neuronavigation in resecting solitary intracerebral contrast-enhancing tumors: a randomized controlled trial. Journal of Neurosurgery, 104, 360–368.PubMedCrossRefGoogle Scholar
  62. Xu, L., Li, J., Zhang, M., Ai, C., & Wang, L. (2008). Chinese authors do need CONSORT: reporting quality assessment for five leading Chinese medical journals. Contemporary Clinical Trials, 29(5), 727–731.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Amrapali Zaveri
    • 1
    • 2
  • Luciana Cofiel
    • 5
  • Jatin Shah
    • 2
  • Shreyasee Pradhan
    • 2
  • Edwin Chan
    • 3
  • Olivier Dameron
    • 4
  • Ricardo Pietrobon
    • 2
  • Beng Ti Ang
    • 1
  1. 1.National Neuroscience InstituteSingaporeSingapore
  2. 2.Duke-NUS Graduate Medical SchoolSingaporeSingapore
  3. 3.Australasian Cochrane CentreSingaporeSingapore
  4. 4.INSERM U936, Universite de Rennes1RennesFrance
  5. 5.Research on Research GroupSao PauloBrazil

Personalised recommendations