Abstract
Background
Quality dimensions are the most important criteria for predicting the success of an information system. The current study aims to evaluate the success of the Iran Electronic Health Record System (SEPAS) based on the DeLone and McLean model for information system success.
Method
This nationwide cross-sectional study was conducted in 2021. Participants were 468 health information management personnel who had working experience with SEPAS. Data were collected using a questionnaire based on the DeLone and McLean model. The validity and reliability of the questionnaire were confirmed. Data were analyzed using SPSS 22 through descriptive and analytic analysis including t-test and ANOVA.
Results
Most participants were female (70.9%) and almost half of the participants mean age was between 30 and 40 years old (49.6%). The total mean of SEPAS success was 3.42 ± 0.53. According to the participants’ perspectives “system quality” was the most influencing factor on SEPAS success. The least influencing factor was SEPAS “benefits”. There was a significant relationship between the mean score of SEPAS success and age (p value = 0.001), Education level (p value = 0.01), and Work experience (p value < 0.001).
Conclusion
The total mean of system success was not acceptable. SEPAS has not been much successful in providing net benefits like provision of electronic services which locate patients in the center and improve the delivery of care to them. It sounds that SEPAS is not stable enough that means crashes sometimes. Hence, considering the required infrastructures for quick response and stability is more critical, especially when healthcare providers are supposed to use the SEPAS.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Background
Electronic Health Record (EHR) are health records residing in an electronic system specifically designed for data collection, storage, and manipulation, and to provide safe access to complete data about patients [1]. EHRs systematically collect lifelong health information about patients’ inpatient, outpatient, and emergency encounters such as medical history, physician orders, nurse notes, vital signs, and Para-clinical investigations.
Considerable capital has been invested in the implementation of EHRs in many developed countries in previous decades [2, 3]. Developing countries, also have taken some steps to employ information technologies including EHR in their healthcare centers. In Iran, efforts to develop infrastructures for and implement EHRs have been part of the Health Ministry IT strategies [4]. Formally, the Ministry of Health in Iran launched a national project as an electronic health records system (SEPAS) in order to develop a national network of health information in 2007. SEPAS is a national electronic health record for all Iranian citizens that collects patients’ data on each patient encounters from all hospitals in a centralized system [5]. Also, the Patients’ data including demographic information, final diagnosis, procedures, para-clinical results, and accounting billing from hospitals’ information systems (HIS) are transferred to SEPAS and stored in the Ministry of Health databases to be used in the future. The software infrastructure instances are installed in several locations as SEPAS nodes, hosted by each medical university in the country. Hospitals which are under the supervision of the associated medical universities exchange health data through their corresponding SEPAS nodes [6, 7]. Management Centre of Statistics and Information Technology of the ministry is accountable for its development. Now, after about 15 years since 2008, Iran has moved from a starting point of no EHR state to a national wide coverage; with a large amount of patients’ information communicated to and stored in SEPAS as a data warehouse for comprehensive implementation of national EHR. However, it is not clear whether this project was successful enough.
The barriers to EHR successful implementation include user resistance, lack of skills, lack of engagement of frontline clinicians, concern for return on investment, lack of administrative and policy support, and the longer time of using EHR in comparison to paper-based [8,9,10]. Thus, to implement successful comprehensive EHR, prior evaluations are necessary. Similar studies evaluating the success of EHR indicated a need for improving the EHR [11,12,13]. Few studies have been conducted for the comprehensive evaluations of EHRs in Iran. Asadi et al. [14] evaluated the SEPAS project and found that the required resources and requirements have not been considered for the project and consequently it has not been able to meet the pre-defined objectives. However, the viewpoints on the success of EHRs have not been extensively sought; while any shortcoming can negatively affect the usage decisions. Thus, there is a gap in the literature in regard to the evaluation of SEPAS in hospitals. Considering the increasing importance of the SEPAS system in Iran and the capital invested in its development by the Ministry of Health, assessment of the success of SEPAS is essential to understand the value and effectiveness of this information system and justify the capital invested in its development and implementation. Thus, the current study aims to evaluate success of Iran EHR system (SEPAS) based on the DeLone and McLean model for information system success.
Method
Study design and setting
This nationwide cross-sectional study was conducted in 2021. Iran includes 31 provinces with 570 public hospitals including 271 teaching hospitals. Only teaching hospitals were included in this study. Convenient sampling was used to select hospitals from some provinces including Fars, Tehran, Khoozestan, Khorasan Razavi, north khorasan, Hamedan, Isfahan, Chaharmahal va Bakhtiari, and Yazd.
Participants
The target population included healthcare employees working with SEPAS. Patients’ information is transferred from HISs to the SEPAS by the healthcare workers especially health personnel in health information management department and personnel in accounting department of the hospitals. SEPAS is now used to store healthcare information of the patients’ encountering hospitals and is not used by healthcare providers including physicians and nurses. Thus, our target population was health information management personnel who had working experience with SEPAS. The sample size was calculated 468 using Gpower software; to estimate the “system success” score, sample size was calculated based on the effect size obtained from [15] which was 0.13. Significance level, power, and accuracy of estimation were considered 0.05, 0.80, and 0.1 respectively. We used Convenience sampling technique in which the subjects are selected based on availability and willingness to participate.
Data collection
Data were collected using a questionnaire based on the DeLone and McLean model, one of the popular and most validated models for information system success [16, 17]. The model proposes six interrelated constructs of information systems success including system quality, information quality, service quality, (intention to) use, user satisfaction, and net benefits (Fig. 1). Information quality is defined as the quality of the information provided by the system (system output). System quality refers to the desirable features/the overall support that a system provides. Intention to use/use is defined as the users’ intention to use or the perceived actual usage of an information system by users to accomplish multiple tasks. User satisfaction refers to the users’ level of satisfaction when using an information system. Net benefits refer to the extent to which an information system contributes to the individuals, organizations, and group success. These variables measure technical success, semantic success, and effectiveness success of an information system. The questionnaire was developed based on the questionnaire used in [11, 17, 18]. Four experts in health information management with at least ten years’ experience in the field confirmed the content validity of the questionnaire after some revisions. The overall reliability of the questionnaire was calculated through Cronbach’s alpha (á = 0.862). Final questionnaire was consisted of 27 questions categorized in six dimensions including system quality (five questions), information quality (six equations), services quality (six equations), (intention to) use (four equations), user satisfaction (two equations), and net benefits (four equations). Each question is rated based on a five-point Likert score from 1 (disagree) to 5 (agree). The questionnaire was distributed face to face by one researcher, or through online questionnaire link via social media groups consisting target members. Distributing questionnaires continued till the targeted sample size was obtained; this process lasted for about 4 months.
Data analysis
Data were analyzed using SPSS.22. Descriptive statistics were used to report the frequency, percentages, and mean for describing the participants’ characteristics as well as SEPAS success dimensions. T-test was used to compare the rate of SEPAS success according to gender; the ANOVA test was used to examine the rate of SEPAS success according to age groups, educational level, and work experience. Where the ANOVA test showed insignificant differences between groups’ variances, the LSD test was used to make pairwise comparisons between groups, and where the ANOVA test showed significant differences between groups’ variances Tamhane’s T2 test was used for pairwise comparisons. The mean score and standard deviation (SD) are used to describe SEPAS success. Since the number of items in each construct is different, to make constructs comparable, we also used this formula [(mean of construct/5) * (100)] to compute the scores out of 100. The total score of SEPAS success was calculated based on the averages of total scores of constructs out of 100. To determine whether the level of SEPAS success based on the quality of each six constructs is acceptable or not, a one-way t-test was used. If at least 75% score (3.75 out of 5) is obtained for each item, the status will be considered acceptable [15].
Ethical consideration
The research is conducted according to the principles stated by the Vice-Chancellorship for Research Affairs of Shiraz University of Medical Science and is approved by the Ethics Review Board of the Vice-Chancellorship for Research Affairs of Shiraz University of Medical Science (Ethical code: IR.SUMS.REC.1399.798).
Results
468 respondents participated in the study. Demographic data of the healthcare employees showed that most of the participants were female (70.9%) and almost half of the participants’ mean age was between 30 and 40 years old (49.6%). Most of the participants had a “bachelor’s science” degree (69%). Moreover, the work experience of most participants (43.6%) in the health information management department was more than ten years (Table 1).
Table 2 shows the descriptive statistics of SEPAS success constructs. The total mean of SEPAS success was 3.34 ± 0.37 (61.9 out of 100). According to the participants’ perspective “system quality” obtained the most score (71.8 out of 100). “Benefits” obtained the least score (60.05 out of 100). In the system quality construct, the highest and the lowest means were regarded as “It has been easy for me to learn how to use SEPAS” (4.04 ± 0.76), and “SEPAS is stable to a satisfactory degree and crashes seldom” (2.78 ± 0.94) respectively. The participants were satisfied with using SEPAS (3.46 ± 0.74). The participants believed that using SEPAS is not much beneficial and based on their viewpoints the most beneficial feature of SEPAS is “improving decision making” (3.33 ± 0.92) and the least beneficial feature was “improving patient care delivery” (2.80 ± 1.04). Among all items “It has been easy for me to learn how to use SEPAS” was the most influencing factor causing SEPAS success while “It allows me to provide feedback about the system and its services” was the least influencing factor.
Investigating the correlations between the mean score of SEPAS success and demographic characteristics of the employees showed that there were significant correlations between the mean score of SEPAS success and gender (0.007), age (p value = 0.001), Education level (p value = 0.023), and Work experience (p value < 0.014). Leven test confirmed that there were homogenous variances between age groups (p value = 0.466). Accordingly, multiple comparisons using the LSD test showed that the mean score of SEPAS success was less in participants < 30 years old. Likewise, the results of the LSD test (Levene p value = 0.104) showed a significant difference between participants with a bachelor’s science degree and those with a master’s science degree, between the mean score of SEPAS success was less in those with master’s or higher degree. There was also a significant difference between work experience 1–5 and work experience ≥ 10 (p value = 0.014) (Table 3).
One sample t-test results are shown in Table 4. Considering p = 0.00, the assumption (H: μ > 3.75) would be rejected. This demonstes that the SEPAS success rate is not acceptable which means at least a 75% score (3.75 out of 5) is not obtained for any of the items.
Discussion
The current national study evaluated the Iran EHR system (SEPAS) based on the DeLone and McLean model for information system success. The respondents believed that the system was not successfully implemented. Although the participants believed that the “System quality” was almost well enough, implementing SEPAS does not sound “beneficial”. Moreover, employees with master’s or higher degrees believed that the SEPAS system was less successful in comparison to those with bachelor’s science degrees. “Ease of using the system” was the most influencing factor causing SEPAS success.
Generally, the health information management personnel hold positive views on how the quality of the SEPAS and the quality of its information. They believed that SEPAS has the required functionality to support the work in question and is easy to use. These results are consistent with the previous studies [19,20,21,22]. For instance, the participants in a study [20] believed that the EMR system provides the required information about patients and they were satisfied with the accuracy of the system. Moreover, another study [22] found that the respondents were largely positive about the quality of the EHR and were satisfied with using it. Similarly, Saghaeiannejad et al.’s [23] study showed that “ease of learning and use” was the most desirable factor according to the users’ responses. However, it sounds like SEPAS is not stable enough which means crashes sometimes. In order to improve system quality considering the required infrastructures is necessary before EHR implementation. The users in Saghaeiannejad et al. [23] research also believed that the “response time” of the system was not desirable. Researchers believe that continuously upgrading and optimizing the IT infrastructure including computers, cables, and the wireless network would probably contribute to good system quality [11]. On the other hand, the SEPAS infrastructures, are guaranteed to provide high bandwidth and secure communication channels. This secure channel is provided using the SHAMS network, a national private health information network. Transferring health data using the SHAMS network guarantees the confidentiality of patients’ data [7]. SEPAS is not used by physicians or nurses currently, but when healthcare providers are supposed to use SEPAS, quick response and stability are more critical. Spending time using EHRs for supporting care delivery constitutes a large portion of physicians’ day [24]. Despite the increased adoption of EHRs, concerns about the adverse consequences of EHRs use on healthcare providers’ satisfaction and burnout are growing [25]. These concerns necessitate developing EHR innovations. Some of these innovations provide automatic data input [26]. SEPAS is now fed through hospital information systems and is not integrated with any other devices including mobile documentation applications. Nowadays, EHR innovations necessitate integrating EHRs with many devices which led to decreased click burden and allows clinicians to spend less time on documentation and more time with patients [26]. On the other hand, these innovations may cause even more crashes if the required infrastructures are not considered in advance.
The respondents reported that SEPAS benefits are almost low in regard to removing paper-based systems limitations, improving patient care, or improving communication among healthcare workers. Bossen et al. [11] found that healthcare staff had a very high expectation of future benefits before full implementation of EHR. Perceived net benefits of systems ultimately drive system success [17]; thus, if users do not feel system benefits may reduce its adoption. According to the health information management personnel, SEPAS has not been much successful in providing electronic services which locate patients in the center and improve the delivery of care to them.
Results also showed that educational level influence the perceived system success. Employees with master's or higher degrees believed that the SEPAS system was less successful in comparison to those with bachelor’s science degrees. This finding is inconsistent with similar researches [27, 28] who indicated that educational level is a technology adoption moderator, the higher the education level the higher is the technology adoption. An explanation for this inconsistency may be the more knowledge of employees with a higher degree about information technologies which has influenced their expectation about an optimal information system.
Although the total mean of system success was in the acceptable range, it was far from the ideal situation (highest score). This is consistent with the Saghaeiannejad et al. [23] findings. Thus, it is important to develop EHRs based on end users’ requirements and expectations rather than the developers’ expectations. The Iran EHR implementation plans at the national level seem ambitious in comparison to other countries; however, it is important to pay attention to the required communication infrastructures, digital health data regulations, and communication standards. Healthcare centers are suggested to implement a local EHR according to their organizational systems, process, and workflows rather than trying to develop a national EHRs which are difficult to integrate with the existing systems. The integration and communication of these local EHRs can be facilitated through the development of communication standards that each healthcare center would comply with.
A limitation of this study is that the assessment was based on the health information management staff, while financial staffs are also the users of the SEPAS whose viewpoints were not investigated. Since financial departments’ staffs have the responsibility of entering data through a similar user interface; it was supposed that they may have similar viewpoints to health information management staffs. Viewpoints of financial departments’ staffs can be addressed in future studies.
Conclusion
We have presented a formative evaluation of a national EHR (SEPAS) based on health information management personnel’s point of view. The total mean of system success was not acceptable. SEPAS has not been much successful in providing net benefits like the provision of electronic services which locate patients in the center and improve the delivery of care to them. It sounds like SEPAS is not stable enough which means crashes sometimes. Hence, considering the required infrastructures for quick response and stability is more critical, especially when healthcare providers are supposed to use the SEPAS.
Availability of data and materials
All data will be presented upon request via contacting the corresponding author.
Abbreviations
- SEPAS:
-
Iranian electronic health record (translation of the Persian phrase)
- EHR:
-
Electronic health record
- EMR:
-
Electronic medical record
References
Ben-Assuli O. Electronic health records, adoption, quality of care, legal and privacy issues and their implementation in emergency departments. Health Policy. 2015;119(3):287–97.
Berner ES, Detmer DE, Simborg D. Will the wave finally break? A brief view of the adoption of electronic medical records in the United States. J Am Med Inform Assoc. 2005;12(1):3–7.
Chantler C, Clarke T, Granger R. Information technology in the English national health service. JAMA. 2006;296(18):2255–8.
Khoshraftarrodi A, Jalal IA, Kashi F, Ahmadi J, Kalantari A, Khosroabadi H, et al. Development of EHR architecture. Vision of EHR. Statistics and Information Technology Management Center of Ministry of Health; 2007. p. 1–54.
Abbasi R, Khajouei R, Mirzaee M. Evaluating the demographic and clinical minimum data sets of Iranian National Electronic Health Record. BMC Health Serv Res. 2019;19(1):450.
Riazi H, Jafarpour M, Bitaraf E. Towards National eHealth Implementation—a comparative study on WHO/ITU National eHealth Strategy Toolkit in Iran. Stud Health Technol Inform. 2014;205:246–50.
Sheikhtaheri A, Tabatabaee Jabali SM, Bitaraf E, TehraniYazdi A, Kabir A. A near real-time electronic health record-based COVID-19 surveillance system: an experience from a developing country. Health Inf Manag J. 2022;18333583221104213.
Gesulga JM, Berjame A, Moquiala KS, Galido A. Barriers to electronic health record system implementation and information systems resources: a structured review. Procedia Comput Sci. 2017;124:544–51.
Attafuah PYA, Abor PA, Abuosi AA, Nketiah-Amponsah E, Tenza IS. Satisfied or not satisfied? Electronic health records system implementation in Ghana: Health leaders’ perspective. BMC Med Inform Decis Mak. 2022;22(1):249.
Kroth PJ, Morioka-Douglas N, Veres S, Pollock K, Babbott S, Poplau S, et al. The electronic elephant in the room: physicians and the electronic health record. JAMIA Open. 2018;1(1):49–56.
Bossen C, Jensen LG, Udsen FW. Evaluation of a comprehensive EHR based on the DeLone and McLean model for IS success: approach, results, and success factors. Int J Med Inform. 2013;82(10):940–53.
Dharma G, Sukadarmika G, Pramaita N. Application of DeLone and McLean methods to determine supporting factors for the successful implementation of electronic medical records at Bali Mandara Eye Hospital. J Appl Sci Eng Technol Educ. 2022;4(2):146–56.
Khajouei R, Abbasi R, Mirzaee M. Errors and causes of communication failures from hospital information systems to electronic health record: a record-review study. Int J Med Inform. 2018;119:47–53.
Asadi F, Moghaddasi H, Rabiei R, Rahimi F, Mirshekarlou SJ. The evaluation of SEPAS national project based on electronic health record system (EHRS) coordinates in Iran. Acta Inform Med. 2015;23(6):369–73.
Safdar R, Ghazisaeidi M, Jebraeily M, Masar E, Shikhtaye M, Farajolah S. Hospital information systems success: a study based on the model adjusted DeLone and McLean in UMSU Hospital. Eur J Exp Biol. 2014;4(5):37–41.
DeLone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst. 2003;19(4):9–30.
Ojo AI. Validation of the DeLone and McLean information systems success model. Healthc Inform Res. 2017;23(1):60–6.
Shim M, Jo HS. What quality factors matter in enhancing the perceived benefits of online health information sites? Application of the updated DeLone and McLean Information Systems Success Model. Int J Med Inform. 2020;137:104093. https://doi.org/10.1016/j.ijmedinf.2020.104093.
Top M, Yilmaz A, Gider Ö. Electronic medical records (EMR) and Nurses in Turkish Hospitals. Syst Pract Action Res. 2013;26(3):281–97.
Zaheer S, Sayed SA. Evaluation of EMR implementation in a private hospital from use’s perspective. GSTF J Nurs Health Care. 2013;1:143–7.
Top M, Gider O. Nurses’ views on electronic medical records (EMR) in Turkey: an analysis according to use, quality and user satisfaction. J Med Syst. 2012;36(3):1979–88.
Tubaishat A. Evaluation of electronic health record implementation in hospitals. Comput Inform Nurs CIN. 2017;35(7):364–72.
Saghaeiannejad-Isfahani S, Saeedbakhsh S, Jahanbakhsh M, Habibi M. Analysis of the quality of hospital information systems in Isfahan teaching hospitals based on the DeLone and McLean model. J Educ Health Promot. 2015;4:5.
Overhage J, McCallieJr D. Physician time spent using the electronic health record during outpatient encounters. Ann Intern Med. 2020;172(3):169–74.
Ehrenfeld JM, Wanderer JP. Technology as friend or foe? Do electronic health records increase burnout? Curr Opin Anaesthesiol. 2018;31(3):357–60.
Guo U, Chen L, Mehta PH. Electronic health record innovations: helping physicians—one less click at a time. Health Inf Manag J Health Inf Manag Assoc Aust. 2017;46(3):140–4.
Riddell WC, Song X. The role of education in technology use and adoption: evidence from the Canadian workplace and employee survey. ILR Rev. 2017;70(5):1219–53.
Abu-Shanab EA, editor. Education level as a technology adoption moderator. In: 2011 3rd International conference on computer research and development; 2011.
Acknowledgements
We acknowledge Farshad Falahati who contributed in data collection. We acknowledge all participants who helped us conduct this research.
Funding
The research was financially supported by the Vice-chancellor for Research Affairs of Shiraz University of Medical Sciences (Grant No. 21498).
Author information
Authors and Affiliations
Contributions
A.B. was the research supervisor. A.B. and F.N. contributed in the study design. S.Z., M.SH., and F.N. contributed in the data collection. S.N. analyzed the data. S.Z. prepared the manuscript draft. All authors have read and approved the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The study is approved by the ethics review board of the Vice-chancellor for Research Affairs of Shiraz University of Medical Science (Ethical code: IR.SUMS.REC.1399.798). All methods were carried out in accordance with relevant guidelines and regulations. Online data gathering through social media let individuals participate based on willingness. A person receiving the questionnaire link could ignore it if he/she was not willing to participate. Thus, following the link, reading the statement explaining the study at the beginning of the questionnaire, and completing the questionnaire are considered as consent to participate. All the participants provided informed consent to participate. This was accepted by the ethics review board.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Bashiri, A., Shirdeli, M., Niknam, F. et al. Evaluating the success of Iran Electronic Health Record System (SEPAS) based on the DeLone and McLean model: a cross-sectional descriptive study. BMC Med Inform Decis Mak 23, 10 (2023). https://doi.org/10.1186/s12911-023-02100-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12911-023-02100-y