Evidence-based medicine self-assessment, knowledge, and integration into daily practice: a survey among Romanian physicians and comparison between trainees and specialists
- 118 Downloads
A gap between the attitude towards evidence-based medicine (EBM), knowledge and awareness has been reported among physicians from different parts of the world. However, no investigation on Romanian physicians is available in the scientific literature. Our study aimed, firstly, to assess EBM awareness and the knowledge used by Romanian physicians, and, secondly, to compare resident trainees with specialists.
Romanian trainee and specialist physicians were invited to participate in this cross-sectional study. The study tool was an online questionnaire designed to explore their awareness, knowledge, usefulness, the attitude in medical documentation, and the use of professional EBM resources. Data were collected by Google Form from January 1st to April 30th, 2017, respecting the responders’ anonymity. Two groups of physicians were investigated as trainees and specialists, respectively. Descriptive statistics (number, percentage, median and interquartile range) was used to describe the survey-related variables. Statistical significance on qualitative data was calculated with the Chi-square test, Fisher’s exact test, or the Z-test for proportions.
Two hundred and 50 physicians participated in this study (68% trainees vs. 32% specialists). In both groups, a significantly high percentage was represented by women as compared to men (trainees 72.4%, specialists 70%). The correct definition of EBM was identified by most respondents (75.6%). Affirmatively, both trainees and specialists always looked at levels of evidence when reading scientific literature, but a small percentage (6.5% trainees and 3% specialists) adequately identified the uppermost types of evidence in the hierarchy. Almost a quarter of the respondents shared the name of mobile EBM resources that they used to support the daily practice. Only six out of the 49 listed mobile resources met the EBM criteria.
The participants proved to have limited knowledge of EBM and a positive attitude towards the concept. They made use of mobile medical resources without understanding which of these were evidence-based.
KeywordsEvidence-based medicine (EBM) Survey Mobile applications Physician
Body mass index
Body surface area
Degree of recommendation
Level of evidence
Randomized controlled trials
Evidence-based medicine (EBM) is the medical practice approach designed for optimizing decision-making by emphasizing the use of evidence supported by systematic and valid medical research. Evidence-based medicine starts from the premise that evidence of the highest quality, yielded from meta-analyses, systematic studies, and randomized clinical trials can offer accurate recommendations concerning individual medical decisions . According to current definitions, EBM aims to render medical decisions more structured and objective, based on the clinical research results [2, 3]. This involves the use of guidelines developed based on results obtained from medical research, in the context of the clinical experience of the physician, and the patient’s desires and beliefs, in order to provide health care for individuals .
Aguirre-Raya et al. reported in 2016 the perception of EBM among medical students, interns, and specialists and reported a high global index of self-perception (75%), but this proved not to be supported by the global index on knowledge (19%) . The gap between the attitude towards EBM and knowledge and awareness has been reported among physicians from different parts of the world (e.g., Saudi Arabia , United Arab Emirates , Ethiopia , Egypt , Japan , Belgium , Norway , France , or India ).
Medical applications that run on the smartphone could be an important source of information for EBM. Mobile medical applications emerged in 2008 and are available through dedicated stores (such as Apple Store, Google Play, Windows Phone Store, and BlackBerry App World) . A study conducted in 2016 showed that just two out of 147 assessed EBM mobile applications running on Android, namely Medscape and DynaMed Plus, proved to be evidence-based .
Little is known about the clinical use of EBM in Romania. The few available articles are reviews [17, 18] or are limited to the evaluation of the knowledge gained after evidence-based training [19, 20, 21]. No articles were identified in the available scientific literature that presented the degree to which Romanian physicians managed to integrate evidence-based medicine into their daily practice.
Our study aimed to assess awareness, information and daily use of evidence-based medicine among Romanian physicians.
A cross-sectional and descriptive-analytical study was conducted from January 1st to April 1st, 2017.
The participants in this study were physicians (medical trainees and specialists, regardless of their specialty) working in healthcare institutions in Romania. The graduates become trainees (or resident physicians) whenever they achieve at least 60% of the maximum score in the national examination (200 multiple choice questions answered within 4 hours). The residency period may vary from three to 7 years, depending on the specialty (e.g., 3 years for Family Medicine, 5 years for medical specialties, and up to 7 years for some surgical specialties such as Neurosurgery). A specialty exam ends the residency period and the trainees can opt not to do this exam and to start a new residency; these participants were considered in our study as trainees.
Section B is available just for the users who choose “smartphone” as the answer in B01. Any other answer in B01 will lead the respondent to Section C. The answer of ten questions in Section A and two questions in Section B, along with the filling of Section C were required for valid submission for respondents who have an smartphone. The majority of the questions in the survey were closed, except for A09, A10, B07, B10, B12, C04 to C07, C10 to C13 that were open-questions (Fig. 1).
The classification systems for the quality assessment of the evidence formulated by the United States Preventive Services Task Force  were used in this questionnaire for the hierarchy of evidence and recommendation degrees.
The development of the survey and its validation were previously reported . The validation of the questionnaire was carried out exclusively for Sections A and B. The items of the questionnaire, which were considered irrelevant, were not reported in this manuscript (Section C, namely C01, C05, C07, C11-C13, Fig. 1).
The online Romanian language version of the self-administered questionnaire, was promoted on professional groups on Facebook (see Additional file 1). All available Facebook groups of trainees and specialists as per January 1st, 2017 were selected for each specific specialty, including mixed groups that comprised physicians from various specialties in the country. A letter of intent, explaining the reason for joining the group and the purpose of the survey, was sent to each group administrator. Twenty-four professional groups consisting of about 75,117 members, with potential overlap (e.g., members who belonged to more groups) were used in order to ask the physicians to participate in this study. An invitation for participation was posted on each group, and a link to a Google Form was provided. At the beginning of the survey, the users were informed about the purpose of the study, the data collection procedure and anonymity (no personal data were collected to allow the identification of the respondent).
A valid response was defined as the absence of an identical entry (identical answers to all questions in the survey) to avoid multiple participation.
The respondents were divided into two groups for analysis, namely: trainees and specialists. The respondents in training were considered trainees, irrespective of whether they were doing the first or second specialty.
Qualitative data were presented as numbers and percentages with associated 95% confidence intervals (provided in squared brackets along with the manuscript)  and were analyzed using the Chi-square test or Fisher’s exact test, as appropriate. Age was tested for normality with the Kolmogorov-Smirnov test, reported as a median and interquartile range, and compared with the Mann-Whitney test. Statistical significance was set at p-value < 0.05.
The mobile applications used and shared by the respondents were assessed to see whether they respected the evidence-based principles. Any apps that met the following three criteria were considered evidence-based medical applications: references supported the provided information, the degree of recommendation was specified, and the level of evidence was present.
Characteristics of the participants
Demographic characteristics and the specialty membership of the respondents
(n = 170)
(n = 80)
Gender, no. (%)
Age, years median (Q1 to Q3) *
27 (26 to 28)
40 (33 to 49)
Work place, no. (%)
Specialty, no. (%)
Mixed specialties c
The graduation year of the respondents varied from 1978 to 2016; most of the trainees graduated from the Faculty of Medicine 5 years prior to the year when the study was conducted (from 2012 to 2016; 77.6% [70.6 to 83.5]).
First year trainee physicians who participated in the study presented the highest percentage (35.8% [28.8 to 43.5]), while the smallest percentage was witnessed by trainees in the 5th (8 participants) or 6th (one participant) year of training.
Evidence-based medicine assessment
Evidence-based medicine knowledge, awareness, and attitudes; numbers (percentage) reflect the results only for data from closed questions
(n = 170)
(n = 80)
(A01) The concept of “evidence-based medicine”? a
I have heard of it, but I do not know what it means …
I know, I understand…, I am not using it…
I know, I understand…, I am using it…
I have never heard of it
(A02) Source of information on EBM (multiple answers allowed)b
(A03) No. of correctly identified definitions c
(A04) Uppermost in the hierarchy of evidence b
Systematic reviews and Meta-analyses
Randomized controlled trials (RCT)
Systematic reviews, Meta-analyses, RCT
(A05) Lowest in the hierarchy of evidence b
In vitro research
Idea, editorial, or expert opinions
An animal research, in vitro research
Idea, editorial, review or expert opinions, animal research, in vitro research
(A06) Level of evidence considered when reading literature a
Never, but I know this classification
I have ignored this classification
I have never heard of it
(A07) Strength of evidence a
Never, but I know this classification
I have ignored this classification
I have never heard of it
(A08) Continuing medical education: reading b
Online free medical articles
Online paid medical articles
Printed free medical articles
Printed paid medical articles
Medical articles available on apps
(A11) Reading scientific articles: preferences a
I do not know the difference
I do not read articles or journals; I prefer books
(A12) The use of EBM b
To know what is new in the medical field
To know the structure of a scientific paper
I have not yet understood the usefulness of EBM
I do not consider EMB in current practice
The majority of respondents declared that they always looked at levels of evidence when reading scientific literature, but fewer than 6% adequately identified the uppermost types of evidence in the hierarchy (Table 2).
The financial aspect (39.2% [33.2 to 45.6]) and accessibility (easy and fast, 14.8% [10.8–19.59]) were the main reasons why the use of free sources of professional information was preferred. Sixty-eight of the respondents (27.2% [21.6–33.19]) pointed out that the accuracy of the information in paid scientific sources was higher as compared to free resources, but without providing any arguments. This result strictly reflects the perception of the respondents since the accuracy was not tested.
Professional application usage on mobile phones
Medical smartphone app usage as reflected by the closed questions
(n = 170)
(n = 80)
(B02) The use of medical apps, yes a
(B03) Reason for using … b
Allowing quick access to medical information
Most colleagues do it
To save time in identifying solutions to clinical problems
(B04) Top five type of medical applications b
(B05) When do you use the apps? b
for each patient, regardless of case
for atypical patients
to stay in touch with medical news
at home, for individual study
(B06) How can apps improve medical care? b
Fast search of scientific literature
Shortest time from consultation to diagnosis and treatment
Encouraging continuous medical documentation …
Availability to be used in any circumstance, even at the bedside …
(B08) Can mobile medical apps reduce healthcare costs? a
Yes, because they decrease unnecessary health service requests
Yes, because they avoid further expensive investigation
They could cut the future costs of healthcare …
(B09) Drawbacks in using medical apps b
incomplete definitions and treatment regimen
the absence of scientific references to support the evidence
an Internet connection is needed to access scientific articles
they are limited to English speakers
the readability level is too low
(B11) EBM medical apps b
Explicit references, LE and DR
Only specified the LE
Only specified the DR
Only specified the references
I do not know mobile medical apps that implement EBM
I have not paid attention to this issue
The results obtained in this study indicated a gap between the respondent’s EBM self-assessment perception and knowledge. This gap was identified in the youngest physicians (trainees) as well as in the specialists, without any differences between these two groups. However, the respondents have a positive attitude towards the EBM concept. The mobile medical resources are universally used by all respondents, but a small number of the listed mobile EBM resources is evidence-based.
Knowledge, awareness, and attitudes
A significantly high percentage of women in the investigated sample reflects the gender distribution of graduates in Romania [25, 26]. A large percentage of the responders were trainees, which could be justified by the method of data collection, the trainees being more active on Facebook. Only 12% of all Romanian Facebook users were aged between 55 and 64 in 2017, while 22% of all Facebook users were aged 35 to 44 . The interest of young physicians in evidence-based medical practice could also explain their participation in the study.
The investigated sample had a similar distribution of demographic characteristics of trainees and specialists regarding their workplace and specialty, with most of respondents having a medical specialty (61.8% of trainees and 71.3% of specialists). The predominance of medical specialties among respondents of EBM surveys was previously reported .
With few exceptions, trainees and specialists showed similar results regarding the EBM self-perception, knowledge, and attitudes towards evidence-based practice. In our sample, a significantly higher percentage of trainees received the information on EBM from their teachers. Similarly, a significantly higher percentage of trainees as compared to specialists were aware of the use of EBM as a source of knowledge and a way to know the structure of a scientific article.
Similar results were previously reported regarding the EBM self-assessment as well as the utility of evidence-based practice and the use of the EBM concept in the daily activity [5, 28]. Our study also reveals disagreement between how the respondents perceived their own EBM knowledge and their actual knowledge (Table 2, A01, A03, A04, A05). The knowledge had a direct impact on the decision-making process and could have a negative impact on clinical decisions. In our study, insufficient critical appraisal skills of a meta-analysis could explain the second place of meta-analysis in the list (Table 2, A04). Kitto et al. reported in 2007, in a small sample of surgeons, that personal experiences, one component of EBM, are seen as the best source of “evidence” .
It is widely recognized that accurate and valid scientific evidence is value-laden , but today’s evidence may turn out to be wrong tomorrow. The number of medical articles published in the scientific literature increases yearly and is also associated with the increasing number of retracted manuscripts  with misconduct as one of the chief causes of withdrawal, a severe issue to evidence-based practice . Affirmatively most respondents consider the level (A06, Table 2) and strength of evidence (A06, Table 2) when they read scientific articles, but no reflection of knowledge related to the place of an article in the hierarchy of evidence (A04 and A05, Table 2) is observed. However, the classification of medical evidence strictly based on the type of the articles is superficial , and the training of physicians must be moved from the type of the manuscript to the quality of the article by assessing the applied methodology used to support the validity of the evidence. A meta-analysis classified as level A in the pyramid of evidence is useless if it proves to be of low or very low quality , and it is recommended not to be considered in medical decisions. The evidence-based practice training curriculum must be adapted to the changing nature of research evidence, to ensure a critical and deep understanding of the scientific literature but not based on a hierarchy according to the type of article.
Medical applications and professional resources
Similarities between trainees and specialists were observed regarding the type of medical apps, regarding the situations when these tools were used, and the perception vis-à-vis to their usefulness in healthcare. There were also differences, though. Compared to specialists, trainees had a significantly more frequent use of medical apps out of curiosity and for individual professional study at home. A more frequent use of the disease aetiology apps by trainees, as compared to specialists, which was identified in our study, could be explained by the trainees’ lack of professional experience. Furthermore, a higher percentage of trainees pointed out the absence of scientific references to sustain the evidence and the non-functional application as the main drawbacks of medical apps. Some of the differences may be related to the specialists drawing on higher clinical expertise and clinical pattern recognition, both of which being parts of an evidence-based practice approach. The absence of scientific references and use of non-functional apps by specialists could be explained by the education received by trainees as well as by their eagerness of trying more applications. A minimal number of the medical apps listed by the respondents met the EBM criteria (6/49).
Medical apps have brought many benefits to the medical field, namely: chiefly faster decisions, with lower error rates, higher quality medical information, accessibility to it , and support in clinical decision-making at the place of treatment . Medical apps have automatic updates as a main feature and allow fast access to up-to-date scientific literature [37, 38]. Medical apps could also employ standard formulas to make calculations and determine risks, such as body mass index (BMI), body surface area (BSA), or adequate drug dosage . More complex mobile apps serve as a diagnostic tool in migraines by identifying the triggering factors; they also facilitate the patient-doctor contact during migraine attacks, and assist the management of treatment strategies . Medical apps can help doctors to offer personalized initial antidepressant drugs according to the patients’ symptoms, conditions and other medications . The use of medical apps has also been increased in specialities where clinical observation is a vital diagnostic tool such as Dermatology, Ophthalmology or Radiology [41, 42, 43]. Furthermore, the usefulness of smartphone apps in telecare was also reported, especially in the long-term management of diseases, with increase in medication adherence and easy patient-doctor communication, especially in the case of initiation of new treatments . Medical apps started to be an important part of clinical medicine, helpful for patients and doctors alike.
Limitations, implications, and perspectives
The main limitation of our study is related to its design. Several issues could be listed, related to the sampling method, data collection, and the content of the survey. Firstly, a non-probabilistic method was used in our study to identify the sample, so the results obtained need to be carefully interpreted. According to the physician medical code, a probabilistic sampling method, such as the simple or stratified by age random method, could be a better solution in order to identify the respondents. Hence, this will closely reflect the population of Romanian physicians, allowing the generalizability of the results. Secondly, the use of Facebook groups as a method to invite the target population to participate in the survey could induce a selection bias, and thus, the investigated sample is not necessarily representative for the Romanian physicians. The largest demographic groups of Facebook users worldwide are represented by 18–27 - year - old women, and 25–34 - year - old men, respectively . In this context, the method used in our study for inviting the participants limits the possibility to reach the senior physicians.
Consequently, the reported results have a limited impact on the population of Romanian physicians, thus the behaviour of young physicians outweighing the actions of senior physicians. The use of the random sampling method by considering all Romanian physicians and sending a printed survey by post or an individual e-mail invitation with recall could engender a more realistic picture of the use of EBM in their daily practice by Romanian physicians. Thirdly, the survey that was used evaluated the respondents’ EBM knowledge and the self-perception behaviour in connection with the use of medical evidence, either printed or online. A more in-depth assessment based on problem-based scenarios would allow a reli able assessment of the EBM knowledge and their application in clinical practice. Furthermore, since the “survey approach is a research strategy, not a research method”, it was intended to evaluate “how things are at a specific time” , and thus had its limitations, such as the lack of details and in-deep analysis, the fact that the respondent could provide a predictable answer, the lack of control over the person who actually accesses the survey link, and the possibility of arbitrary answers . A further limitation could be grouping, definition and subsequent analysis of trainees, who were defined as those currently in training at the time of the questionnaire completion, irrespective of whether previously being trainees or having completed one or more specialties. However, since just only 14 participants were in this situation, this was not expected to significantly influence the results.
Despite its limitations, our study has had several characteristics of value. Firstly, our results provided a snapshot regarding EBM self-assessment and knowledge in the group of respondents. Our result supports the need for evidence-based practice training at the level of undergraduate students, trainees as well as specialists. The onsite or online courses, even during scientific meetings and conferences, with the translation of EBM information/knowledge into problem-based learning, could ensure a better understanding of the concepts. The shift from the use of evidence based on the type of the article to the use of medical evidence based on their quality is a must and could be done through more thorough EBM training. Integrating EBM in the daily health care practice could serve as a supplementary resource for knowledge, in addition to guidelines, which are very helpful for clinical decision-making whenever a specific case not covered by the clinical guidelines needs to be managed.
Most of the participants in our study were able to identify the correct definition of evidence-based medicine but failed to recognize the levels of evidence correctly. The participants’ attitude towards EBM was positive. Mobile medical apps were used in the daily clinical activity, but the respondents could not differentiate between the apps, which comply and those which do not comply with EBM principles. The changing nature of research evidence supports the move beyond the EBM training to quality assessment regardless of level, grade or evidence.
RDC conceived the idea for the study, conducted the survey design, and disseminated the study. She also did the recruitment of participants via Facebook, and data collection wrote up the results and produced the first draft of the manuscript. AEB assisted in the writing up and interpretation of the results and provided a review of the manuscript and revisions. SDB supervised the survey design, conducted the analysis, wrote up the results and provided the review of the manuscript and other revisions. All authors read and approved the final manuscript.
This research was supported financially by “Iuliu Hațieganu” University of Medicine and Pharmacy, Cluj-Napoca, Romania, through PCD grant no. 5200/16/01.03.2017. The funding for open access publication has been supported by the authors.
Ethics approval and consent to participate
The study has been approved by the Ethics Committee of “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca (no. 207/06.05.2017) and is consistent with the ethical guidelines of the 1975 Declaration of Helsinki. The consent for participation was considered to have been given as each participant agreed to fill in and submit the survey, while the anonymity of the participants was guaranteed. Contact details of the principal investigator were provided.
Consent for publication
Participants were informed that the study was part of PhD research and thus the results may be published in scientific journals. None of the respondents was asked to provide any personal identifiable information, to guarantee their anonymity.
None of the authors has any conflicts of interest to declare.
- 6.Alshehri AA, Al-Khowailed MS, Alnuaymah FM, Alharbi AS, Alromaihi MS, Alghofaili RS, Al-Maddallah WS, Alqattan SJ, Alyahya MA. Knowledge, attitude, and practice toward evidence-based medicine among hospital physicians in Qassim region, Saudi Arabia. Int J Health Sci (Qassim). 2018;12(2):9–15.Google Scholar
- 14.Ambulkar R, Ranganathan P, Karthik V, Divatia J. Evidence-based medicine: a survey among perioperative health care professionals in India. J Anaesthesiol Clin Pharmacol. 2017;33(4):487–92.Google Scholar
- 16.Capraş RD, Bolboacă SD. An evaluation of free medical applications for android smartphones. Appl Med Inform. 2016;38(3–4):117–32.Google Scholar
- 17.Anastasiu M, Strâmbu V, Popa F. Evidence-based medicine, provocare conceptuală sau viitorul cotidian al practicii medicale? Chir. 2007;102(5):527–30.Google Scholar
- 18.Sgarbură O, Tomulescu V. Popescu. Chirurgia bazată pe dovezi: abordarea contemporană a practicii medicale. Chir. 2007;102(2):155–60.Google Scholar
- 19.Bolboacă S, Achimaş CA. Assessment of a mini-course in evidence-based medicine. Appl Med Inform. 2005;16:7–14.Google Scholar
- 20.Bolboacă SD, Achimaş Cadariu A, Ţigan S. Distance Learning Strategy: Continuing Medical Education on Evidence-Based Medicine - Implementation at Cluj County. Proc 11th World Congress Internet Med. 2006.Google Scholar
- 21.Marta MM, Bolboacă SD, Jäntschi L. Self-directed Training in an Evidence-Based Medicine Experiment. In: Hippe ZS, Kulikowski JL (Eds): Human-Comput Syst Interact. Adv Intell Soft Comput, Springer, Berl, Heidelberg, 2009;60:193–204.Google Scholar
- 22.Grade Definitions - US Preventive Services Task Force. 2017. Available at: https://www.uspreventiveservicestaskforce.org/Page/Name/grade-definitions (Accessed 9 Dec 2017).
- 23.Capraș RD, Drugan TC, Bolboacă SD. Development and Validation of a Questionnaire to Assess Evidence-Based Practice. Proceedings of the 6th IEEE International Conference on E-Health and Bioengineering – EHB. 2017;6:129–32.Google Scholar
- 27.Statistics F. Top three county with most users [Statistici Facebook]. Top 3 orase cu cei mai multi utilizatori [Internet]. re7consulting. 2019; Available at: https://www.re7consulting.com/statistici-facebook-romania-infografic/ (Accessed 4 Feb 2019).
- 34.Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P. Schünemann HJ. GRADE Working Group GRADE: an emerging consensus on rating quality of evidence and strength of recommendations BMJ. 2008;336:924.Google Scholar
- 35.Lee VC. Mobile devices and apps for health care professionals: uses and benefits. PT. 2014;39(5):356–64.Google Scholar
- 37.Abbasgholizadeh Rahimi S, Menear M, Robitaille H, Légaré F. Are mobile health applications useful for supporting shared decision making in diagnostic and treatment decisions? Glob Health Action. 2017;10(3):38–41.Google Scholar
- 40.Man C, Nguyen C, Lin S. Effectiveness of a smartphone app for guiding antidepressant drug selection. Fam Med. 2014;46(8):626–30.Google Scholar
- 42.Hogarty DT, Hogarty JP, Hewitt AW. Smartphone use in ophthalmology: what is their place in clinical practice? Surv Ophthalmol. 2019. https://doi.org/10.1016/j.survophthal.2019.09.001.
- 45.Distribution of Facebook users worldwide as of April 2019, by age and gender, 2019 Available at: URL https://www.statista.com/statistics/376128/facebook-global-user-age-distribution/ (Accessed 5 June 2019), .
- 46.Denscombe M. The good research guide: for small-scale social research projects. Buckingham: Open University Press; 1998.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.