Despite the prevalent use and advantages of information systems in hospitals, some have failed to meet their predefined objectives. Surgery information system (SIS) is a sub-system of a hospital information system. Its effective and efficient operation could enhance patient care in the busy environment of operating rooms with multiple tasks. The objective of this study was to evaluate the effectiveness and efficiency of SIS in three educational hospitals.
Data were collected using a questionnaire completed by 82 users of SIS. This questionnaire contains three parts: 1) participants’ demographic information, 2) questions regarding the efficiency of SIS, and 3) questions about its effectiveness. An independent sample t-test was used to compare the efficiency and effectiveness among systems. Chi-squared and Fisher tests were used to determine the relationship between the participants’ demographics and efficiency and effectiveness as well as the relationship between efficiency and effectiveness.
About 23% of the participants rated the system’s efficiency as low, 29% as medium, and 48% as high. Besides, 24% of the participants considered the effectiveness of the system as low, 31% as medium, and 45% as high. There was a significant correlation between the efficiency and effectiveness of SIS (p ≤ 0.0001).
Based on the perspective of most participants (44%)the efficiency and effectiveness of both surgery information systems were acceptable. The results suggest that these systems should be designed in a way that facilitate user’s interaction and reduce the time takes to complete tasks. The results could be useful for developing and designing an efficient and effective system.
Hospital information systems (HIS) have been developed with the aim of collecting, storing, processing, and providing access to patients’ information through computers and communication tools. They are used for establishing communication among healthcare providers, managing patients’ information, and providing better services to patients . An efficient and effective system is necessary to reach the goals of a health care organization. According to international standard organization (ISO), efficiency and effectiveness are two determinants of system acceptability. Effectiveness is the accurate and successful completion of the system’s goals by a user, and efficiency is the extent of effort and resources accurately and completely used to achieve these goals .
Many studies have shown [3,4,5] the efficiency and effectiveness of surgery information systems in reducing errors by improving legibility, completeness, and accuracy of information required for accurate decision making by physicians and doing their tasks.
Evidence shows that [6,7,8,9] lack of efficiency and effectiveness not only causes errors but also increases adverse effects and clinical risks for patients. Furthermore, the system will be inefficient, and clinical expenses will be incurred if the tasks are not completed on time. Therefore, an efficient and effective system can enhance patients’ care and safety.
Surgery information system (SIS) is a subsystem of a hospital information system that is used in operating rooms. Since the occurrence of errors in operating rooms is inevitable and has adverse consequences for patients, the surgery team needs an efficient and effective system to provide accurate and efficient information and improve patients care safety [10,11,12].
So far, no study has reported on the evaluation of the effectiveness and efficiency of surgery information systems. The majority of the studies conducted on surgery information systems have addressed topics such as the management of patient’s care , quality assurance, the management of waiting list in surgery information systems , the development of mobile-based surgery information system  and its architecture . The aim of this study was to evaluate the effectiveness and efficiency of the surgery information system from the viewpoints of its users.
This study was conducted in 3 hospitals affiliated to Kerman University of Medical Sciences (Shafa, Afzalipoor, Bahonar) in the fall of 2018. Currently, the Tirazheh surgery information system is used in two hospitals, and Peyvand Dadeh is used in the other hospital. Of the 136 users with active usernames and passwords, 82 use the system actively. All these 82 users were invited to participate in the study.
Surgery information system
The surgery information systems evaluated in this study were the subsystems of two health information systems developed by Tirazheh and Peyvand Dadeh that are currently used in 236 and 67 educational hospitals in Iran, respectively. These systems have various parts, including patient profiles, patient lists, patient status, new surgeries, requests for services, blood requests, service records and audio records, service results, inpatient appointments, patient transfers, and delivery information. Using these systems, users can undertake activities such as transferring patients from the waiting lists to clinical wards and entering the information of services provided to the patient. Also, users can order services from other wards and see the results of the provided services.
The data was collected using a questionnaire that was developed by the researchers based on the review of usability questionnaires  of the system as well as a previous study . This questionnaire was reviewed by three medical informaticians, one health information management specialist, and one operating room technician (Additional file 1). The questionnaire was modified based on the specialists’ comments and its content validity was confirmed. In designing this questionnaire, three questions from the previous study, and two questions from the usability questionnaire were used for the effectiveness section. as well as two questions from the previous study and two questions from the usability questionnaire were used for the efficiency section. In addition, twelve of the questions were designed by researchers. The reliability of the questionnaire was confirmed by calculating the internal consistency with a Cronbach’s alpha of 95%. The questionnaire was composed of three parts: 1) Demographic information such as sex, education, work experience, computer skills, the experience of working with a surgery information system, daily use of the surgery information system, and the experience of using the traditional system. 2) Specific questions for evaluating the effectiveness of the surgery information system (16 questions, one of which related to the overall effectiveness of the system). 3) Specific questions for evaluating the efficiency of the surgery information system (5 questions). We used a 9 point Likert scale (1 = the lowest level of agreement to 9 = the highest level of agreement) to answer the questions. Zero was assigned to the questions with no answer. Effectiveness was assessed based on user’s perceptions of achieving system goals while performing tasks. Efficiency was measured based on the users’ perceptions of reducing the time required to perform tasks.
Data were collected from the hospitals under study by the researcher after obtaining the required license issued by the research deputy of the faculty. Printed questionnaires were distributed by one of the researchers among the participants after explaining the purpose of the study. Verbal consent was obtained from the participants before the completion of the questionnaires. To ensure the completeness of all questionnaires, one of the researchers personally collected the questionnaires in the same day. In the case that the users did not have enough time to complete the questionnaires on the same day, the researcher visited the hospital the next day and asked them to cooperate.
Data were analyzed using SPSS 22. We used independent sample t-test to compare the efficiency and effectiveness of the two systems. Chi-square test and Fisher test were used to investigate the relationship between demographic information and efficiency and effectiveness as well as to determine the relationship between the efficiency and effectiveness of the system. The anchors of the questionnaire were reduced to three categories for easy data analysis (0–3: low, 3.1–6: average, 6.1–9: high) . Moreover, the efficiency and effectiveness of the surgery information system were considered as acceptable if the total score of more than 70% was obtained. Likewise, it was considered as marginal with a score between 50 and 70%, and as not acceptable with a score of less than 50% .
The demographic information of the participants is shown in Table 1. Of the 82 users who participated in the study, 85% were female and 92% had an academic education degree. Forty percent of the participants had 6–15 years of work experience. In addition, 60% of the participants had less than 5 years’ experience of working with a surgery information system, and 58% use the system for less than an hour per day. More than 81% of the users had the experience of working with the traditional system.
Table 2 shows the mean efficiency and effectiveness of the surgery information systems based on their vendors. In general, there was no significant difference between the efficiency and effectiveness of the surgery information systems between Tirazheh and Peyvand Dadeh (p > 0.05).
The rating of the efficiency and effectiveness of the surgery information systems based on demographic information of the participants is shown in Table 3. There was no meaningful correlation between any of the participants’ demographic information (gender, work experience, education, experience of working with computer, computer skills, experience of using a surgery information system, daily use of the surgery information system, and experience of using the traditional system) with efficiency and effectiveness of the surgery information systems (p > 0.05).
In general, about 23% of the participants considered the efficiency of the surgery information systems as low, 29% considered it as medium, and 48% regarded it as high. Besides, 24% of the participants considered the system effectiveness as low, 31% considered it as medium and 45% regarded it as high.
In response to the overall question comparing the effectiveness of the surgery information system with the traditional system, 57% (n = 47) of the participants believed the surgery information system is far more effective. In addition, 45% (n = 37) of them rated the learnability of the system in a short period at a medium level. The results also showed that 56% (n = 46) of the participants believed that the surgery information system can increase the quality of documents and the confidentiality of information. Likewise, 51% believed that using the system would increase the accuracy in recording the information and save the users time (Table 4).
According to the perspective of 44% of the participants who believed that the efficiency of the system was not acceptable and 22% of those who rated it as marginal and 10% of those who considered it as acceptable, the effectiveness of the system was also the same. There was a significant relationship between the efficiency and effectiveness of the surgery information system (p ≤ 0.0001) (Table 5).
According to the perspectives 44% of participants in this study, the efficiency and effectiveness of the surgery information system were acceptable. Also, there was no significant difference between the efficiency and effectiveness of the surgery information system in the two companies. About 30% of users reported that they can learn how to use the system in a short period of time, though they need some training to use it successfully. Moreover, more than 50% of them claimed that using this system can improve the confidentiality and accuracy of patient’s information as well as the completion of documents. The findings of the present study showed that the users irrespective of their age, sex, education, experience of working with computers, computer skills, and experience of working with surgery information systems have the same viewpoint about the efficiency and effectiveness of the system.
In our study, 44.7% of the participants rated the system’s effectiveness as high and 48.5%of them rated its efficiency as high, as well as 44% of participants believed that the system effectiveness and efficiency were acceptable. In this regard, Meulendijk  evaluated a decision support system and showed that users performed their tasks better using the system compared to traditional method, but they sometimes made mistakes when working with the system. Furthermore, due to the users’ unfamiliarity with the user interface, it took more time to get tasks done by the system, leading to low efficiency. Hence, participants indicated that the system was marginally acceptable. Findings of a review study  that evaluated three aspects (effectiveness, efficiency, and satisfaction) of the clinical decision making support system showed a positive effect of this system on efficiency and effectiveness. Also, Treadwell study  assessed the usability of the personal digital assistant system for clinical examinations. According to the results, the effectiveness of this system was not different in the effectiveness of the paper-based system, but its efficiency was higher. Farrahi in a study  measuring the relationship between user interface problems of an admission, discharge, and transfer (ADT) module with usability attributes, found that the efficiency and effectiveness of that system were unacceptable. This was due to the user interface issues such as unclear function of some keys, lack of “help” option, and the existence of hidden menus in the system. These issues prevented a successful interaction of the users with system wasted their time. Findings of Sheehan study  also indicated that quality of a system interface can influence the duration of work and consequently the system performance. Moreover Guo study  showed that system interface problems reduce the efficiency because users have to devote more time to fulfill a task.
The results of the study showed a significant relationship between the efficiency and effectiveness of surgery information systems. The findings of the present study are in line with the results of the study conducted by Erik  concerning the relationship between the three usability attributes (efficiency, effectiveness, and satisfaction). Findings of his study also showed a meaningful correlation between the efficiency and effectiveness attributes. Ramadan , in a study evaluating the usability of geographic information system (GIS), showed that although the effectiveness score was higher than the efficiency score, consistent with our results, there was a significant relationship between these two attributes. In a study conducted by Georgsson  participants who did their tasks successfully were also quicker in performing them.
In our study, 51% of the participants evaluated high rate of the use of the surgery information system to increasing the precision of recording patients’ information. Increased accuracy in data recording reduces errors and re-work, which can decrease the time required to perform tasks and improve the system efficiency. Consistent with this finding, Maat study findings  showed that after performing the computerized physician order entry (CPOE) in the intensive care unit (ICU), the glucose calculation accuracy increased. Use of this system, in comparison with the manual computation, reduced the computation time.
In the present study, about 48% of the users indicated that the physicians’ errors reduced highly when using the system. Also, 34% of the participants believed that this system improved patient safety up to a medium level This finding is consistent with the results of a study by Hsieh  in which the use of mobile electronic medication administration record (ME-MAR) system reduced medication errors by nurses and consequently increased patient safety. About 37% of our study participants mentioned that this system could improve patient satisfaction at a moderate level. This finding is congruent with the results of Yilmaz’s study that investigated the use of an architectural system for the development of smart information systems .
Based on the results, about 35% of the participants rated the system’s ease of use as high. Since system ease of use saves the time users spend on the learning and using a system, it can affect the efficiency of that system. Therefore, the design of such systems should be simplified in their development phase to improve their efficiency. In this regard, a study conducted by Lee  regarding the efficiency of the nursing care planning system showed that ease of use of the system improved efficiency and satisfaction of users. Furthermore, the study by Khajouei  also showed that physicians were satisfied with CPOE ease of use and its effect on medication safety, workflow, and efficiency. Likewise, nurses also had a positive attitude towards it. In the present study, the 45%of the participants mentioned their need for training to use the system. This may not be the case for other systems. For example, Alnasser  in a study which investigated the usability of Twazon Arabic Weight Loss App showed that a large number of users did not need to learn the app before using it. The majority of participants in this study believed that the surgery information system improves communication among specialists (40%) and clinical wards (55%). Since the aim of this communication is to provide care for the patients, the facilitation of communication can improve patient care . Sturzlinger , in a study concerning the efficiency and effectiveness of CPOE with integrated decision support systems, highlighted the improvement of communication and patient care after implementation of this CPOE. Georgiou  in a study revealed that the use of CPOE and clinical decision support system (CDSS) reduced the turnaround time for laboratory test results and improved patient care as a result.
Fifty-six percent of the participants in this study noted that surgery information systems increase the confidentiality of information. This finding is in line with the results of Dale’s study  on the investigation of the privacy, confidentiality, and security of information in electronic information systems. Participants in our study asserted that surgery information systems increase the speed of completing documents. This finding is consistent with the results of previous studies [9, 35].
Implications of the study
Despite the existence of different studies on surgery information systems, no previous study has investigated the efficiency and effectiveness of this system. This research is the first study examining the efficiency and effectiveness of two surgery information systems used in Iranian hospitals. The methodology adopted in this study helped to investigate the opinions of real users concerning the efficiency and effectiveness of surgery information systems. Our findings suggest that policymakers, designers and developers of information systems select and develop efficient and effective systems that help users reach out the system’s goals such as increasing patient safety and satisfaction, reducing task completion time, improving accuracy of patient information and reducing errors rate.
This study had three limitations. First, the efficiency and effectiveness of the system were evaluated in three university hospitals. More accurate results can be obtained by choosing a bigger sample size form a broader environment. However, this research was carried out on two surgery information systems that are currently used in 303 university hospitals in Iran. Second, since the study was conducted on an already implemented system, it was not possible to compare the effectiveness and efficiency of the surgery information system with the traditional system. Yet, the findings of this study can be used by subsequent studies that plan to compare efficiency and effectiveness of the systems before and after implementation. Third, this study examined the efficiency and effectiveness of two surgery information systems (Tirazheh and Peyvand Dadeh). Although the surgery information systems developed by different vendors may have different efficiency and effectiveness based on each questionnaire component, due to the following reasons this may not have a major impact on our findings: a) the efficiency and effectiveness of these two systems were not significantly different, b) since these two systems have the same purpose and are designed for the same goal, they are similar in terms of interface features and capabilities and c) this study was conducted in three teaching hospitals affiliated with the same university, hence, they have almost similar staff composition, and the technical knowledge of their system users is similar.
In the present study, the efficiency and effectiveness of the surgery information systems were acceptable. Also, the efficiency and effectiveness of the two surgery information systems in the two companies (Tirazheh and Peyvand Dadeh) were similar. Based on users’ perspectives, the selection and development of information systems should focus on the aspects that improve the successful interaction of the users with the systems and of the system and reduce the time required to perform tasks This study provided insights for healthcare policy-makers, and designers and developers of information systems to select and develop efficient and effective systems.
Availability of data and materials
The data generated and analyzed during this study are available from the corresponding author on reasonable request.
Computerized physician order entry
Hospital information system
International standard organization
Surgery information system
Geographic information system
Mobile electronic medication administration record
Admission, discharge, and transfer
Intensive Care Unit
Electronic Health record
Clinical Decision Support System
Ahmadian L, Dorosti N, Khajouei R, Gohari SH. Challenges of using hospital information systems by nurses: comparing academic and non-academic hospitals. Electron Physician. 2017;9(6):4625–30.
Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc. 2016;23(1):5–11.
Hardenbol AX, Knols B, Louws M, Askari M, Meulendijk M. Usability aspects of medication-related decision support systems in the outpatient setting: a systematic literature review. Health Informatics J. 2020;26(1):72–87.
Chunmei R, Hualing H, Haihua Z. Design and application of nursing CDSS based on structured EMR. Stud Health Technol Inform. 2018;250:238–9.
Maat B, Rademaker CMA, Oostveen MI, Krediet TG, Egberts TCG, Bollen CW. The effect of a computerized prescribing and calculating system on hypo- and hyperglycemias and on prescribing time efficiency in neonatal intensive care patients. JPEN J Parenter Enteral Nutr. 2013;37(1):85–91.
Stürzlinger H, Hiebinger C, Pertl D, Traurig P. Computerized Physician Order Entry - effectiveness and efficiency of electronic medication ordering with decision support systems. GMS Health Technol Assess. 2009;5:Doc07–Doc.
Nitrosi A, Bertolini M, Notari P, Botti A, Ginocchi V, Tondelli G, et al. Efficiency and effectiveness of an innovative RIS function for patient information reconciliation directly integrated with PACS. J Digit Imaging. 2013;26(3):412–8.
Piva E, Sciacovelli L, Zaninotto M, Laposata M, Plebani M. Evaluation of effectiveness of a computerized notification system for reporting critical values. Am J Clin Pathol. 2009;131(3):432–41.
Hariyati RTS, Yani A, Eryando T, Hasibuan Z, Milanti A. The effectiveness and efficiency of nursing care documentation using the SIMPRO model. Int J Nurs Knowl. 2016;27(3):136–42.
Makary MA, Sexton JB, Freischlag JA, Millman EA, Pryor D, Holzmueller C, et al. Patient safety in surgery. Ann Surg. 2006;243(5):628–35.
Shen-Yuan T, Chien-Liang C, Lun-Ping H, editors. Development of mobile-based intelligent surgery information system. 2015 11th international conference on heterogeneous networking for quality, reliability, Security and Robustness (QSHINE); 2015. p. 19–20. Aug. 2015.
Nouei MT, Kamyad AV, Soroush AR, Ghazalbash S. A comprehensive operating room information system using the Kinect sensors and RFID. J Clin Monit Comput. 2015;29(2):251–61.
Cristóvão R, Gomes P. Management of waiting list for surgery information system. In the European conference on information systems management. University of Ghent (p. 579). Belgium: Academic Conferences International Limited; 2011.
Niiranen S, Välimäki A, Yli-Hietanen J. Experiences from the architectural migration of a joint replacement surgery information system. Biomed Inform Insights. 2008;1:1–5.
Garcia A. UX research | standardized usability questionnaire. 2013.Accessed 13 Jun 2020; Available at: https://arl.human.cornell.edu/linked%20docs/Choosing%20the%20Right%20Usability%20Questionnaire.pdf.
Sadoughi F, Aminpour F. A review on the evaluation methods of health information systems. Iran J Med Educ. 2011;10(5):1077–86.
Ghazisaeedi M, Shahmoradi L, Ranjbar A, Sahraei Z, Tahmasebi F. Designing a Mobile-based self-care application for patients with heart failure. J Health Biomed Inf. 2016;3(3):195–204.
Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact. 2008;24(6):574–94.
Meulendijk MC, Spruit MR, Drenth-van Maanen AC, Numans ME, Brinkkemper S, Jansen PAF, et al. Computerized decision support improves medication review effectiveness: an experiment evaluating the STRIP Assistant's usability. Drugs Aging. 2015;32(6):495–503.
Knols B, Louws M, Hardenbol A, Dehmeshki J, Askari M. The usability aspects of medication-related decision support systems in the inpatient setting: a systematic review. Health Informatics J. 2020;26(1):613–27.
Treadwell I. The usability of personal digital assistants (PDAs) for assessment of practical performance. Med Educ. 2006;40(9):855–61.
Farrahi R, Rangraz Jeddi F, Nabovati E, Sadeqi Jabali M, Khajouei R. The relationship between user interface problems of an admission, discharge and transfer module and usability features: a usability testing method. BMC Med Inform Decis Mak. 2019;19(1):172.
Sheehan B, Lee Y, Rodriguez M, Tiase V, Schnall R. A comparison of usability factors of four mobile devices for accessing healthcare information by adolescents. Appl Clin Inform. 2012;3(4):356–66.
Guo J, Iribarren S, Kapsandoy S, Perri S, Staggers N. Usability evaluation of an electronic medication administration record (eMAR) application. Appl Clin Inform. 2011;2(2):202–24.
Frøkjær E, Hertzum M, Hornbæk K, Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Proceedings of the SIGCHI conference on Human Factors in Computing Systems; 2000.
Ben Ramadan AA, Jackson-Thompson J, Schmaltz CL. Usability Assessment of the Missouri Cancer Registry's Published Interactive Mapping Reports: Round One. JMIR Hum Factors. 2017;4(3):e19–e.
Hsieh S-H, Hou IC, Cheng P-H, Tan C-T, Shen P-C, Hsu K-P, et al. Design and implementation of web-based mobile electronic medication administration record. J Med Syst. 2010;34(5):947–58.
Yılmaz Ö, Erdur RC, Türksever M. SAMS--a systems architecture for developing intelligent health information systems. J Med Syst. 2013;37(6):9989.
Lee T-T, Lee T-Y, Lin K-C, Chang P-C. Factors affecting the use of nursing information systems in Taiwan. J Adv Nurs. 2005;50(2):170–8.
Khajouei R, Wierenga PC, Hasman A, Jaspers MWM. Clinicians satisfaction with CPOE ease of use and effect on clinicians' workflow, efficiency and medication safety. Int J Med Inform. 2011;80(5):297–309.
Alnasser A, Kyle J, Alkhalifah A, Marais D. Relationship between evidence requirements, user expectations, and actual experiences: usability evaluation of the Twazon Arabic weight loss app. JMIR Hum Factors. 2018;5(2):e16.
Kurahashi AM, Stinson JN, van Wyk M, Luca S, Jamieson T, Weinstein P, et al. The Perceived Ease of Use and Usefulness of Loop: Evaluation and Content Analysis of a Web-Based Clinical Collaboration System. JMIR Hum Factors. 2018;5(1):e2–e.
Georgiou A, Lang S, Rosenfeld D, Westbrook JI. The use of computerized provider order entry to improve the effectiveness and efficiency of coagulation testing. Arch Pathol Lab Med. 2011;135(4):495–8.
O’Brien DG, Yasnoff WA. Privacy, confidentiality, and security in information systems of state health agencies. Am J Prev Med. 1999;16(4):351–8.
Hsieh H-Y, Henker R, Ren D, Chien W-Y, Chang J-P, Chen L, et al. Improving effectiveness and satisfaction of an electronic charting system in Taiwan. Clin Nurse Spec. 2016;30(6):E1–6.
The authors thank the people who participated in this study.
No funding was obtained for this study.
Ethics approval and consent to participate
This study was approved by the ethics committee of Kerman University of Medical Sciences (IR.KMU.REC.1397.536). The participation in the study was voluntary and the participants could refuse to receive, fill out the questionnaire or return it to the researchers. Hence, only verbal consent for participation was taken from the participants, after explaining the objectives of the study, emphasizing the confidentiality of their information, and giving assurance that no information about participants will be provided in the research report or paper. Filling out and returning the questionnaires was considered as their agreement for participation. These were documented in the proposal delivered to the ethics committee and were confirmed by this committee.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Abbasi, F., Khajouei, R. & Mirzaee, M. The efficiency and effectiveness of surgery information systems in Iran. BMC Med Inform Decis Mak 20, 229 (2020). https://doi.org/10.1186/s12911-020-01236-5
- Surgery information system