Keywords

5.1 Introduction

The International Organization for Standardization (ISO) defines quality in general as the ability to meet all the expectations of the purchaser of goods or services or, in other words, as the degree to which a set of inherent characteristics fulfills requirements, where “requirements” means needs or expectations.

In Sect. 1.3, we already discussed the requirements of various stakeholder groups and their expectations on the information system. In this chapter, we will now discuss in more detail the various aspects of the quality of a health information system. Assessing the quality of health information systems using quality characteristics, and maintaining this quality, is one of the tasks of managing information systems (Fig. 5.1).

Fig. 5.1
A photo of a consultant physician in a ward, who wears a stethoscope, is staring at the paper. Some people working on computers can also be observed.

Health information systems constitute an essential part of providing good health care. Consultant physician reviews data on a ward

Full size image

After reading this chapter, you should be able to

  • describe different quality characteristics of the management of the information system in a health care facility,

  • describe different quality characteristics of the information system of a health care facility, and

  • describe the steps of systematically evaluating quality characteristics of the management of the information system or of the information system itself.

5.2 Quality of Management of Information Systems

In Chap. 4, we discussed the tasks of strategic, tactical, and operational management of information systems and the related organizational structures of information management. We also introduced IT governance and Information Technology Service Management (ITSM). We will now use a top-down structure when discussing the quality of management of information systems, starting with the quality of IT governance and finishing with the quality of ITSM.

5.2.1 Quality of IT Governance

IT governance is part of the overall management of a health care facility. It aims at providing appropriate organizational structures for managing the information system in such a way that it creates value for stakeholders.

To assess the quality of IT governance, we can assess the following aspects:

IT governance structures should be established that enable the management of information systems to create value for stakeholders and minimize risks related to the information systems (Sect. 4.6.1). For example, responsibilities for strategic, tactical, and operational management need to be clearly assigned to organizational units (such as the Information Management Department, Sect. 4.6.3) or roles (such as the chief information officer (CIO), Sect. 4.6.2). IT governance should be based on established best practice frameworks and standards, such as COBIT (Control Objectives for Information and Related Technology). COBIT is an international framework for IT governance that defines a set of generic processes for the management of IT in an organization for each of these processes. COBIT defines objectives, key activities, inputs and outputs, and performance measures. Processes defined by COBIT [1] include managing the strategic information management plan, managing project portfolios, IT risk management, IT change management, ITSM, or IT operation.

In addition, enough resources should be provided for IT staff, IT infrastructures, and application components so that the information system can best meet the business demands. For example, the annual IT budget should be sufficiently high to hire qualified staff to manage the information system.

Finally, the health information system should be operated in such a way that all information management laws of the specific country are fulfilled. Different laws must be fulfilled in every country, such as laws regarding data protection, data interchange, IT security, or health statistics. These laws must be taken into account by management of information systems.

5.2.2 Quality of Strategic Management of Information Systems

Strategic management of the information system deals with the information processing of a health care facility as a whole. It depends on the facility’s business goals and must translate these into an appropriate strategic information management plan.

To assess the quality of strategic management of information systems, we can look at the following aspects:

First, a strategic information management plan should exist and should be updated regularly. It should be closely and visibly aligned with the business strategy and business goals of a health care facility. This means that the outcome of managing information systems, i.e., the information system itself, should clearly support the business goals of the health care facility. For example, the resulting health information system should support business goals such as high-quality care of chronically ill patients, participation in cross-institutional clinical research, or attracting patients from neighboring regions.

In other words, information logistics should be possible in a way that it supports all intended business processes and functions and fulfills the need of the various stakeholder groups (physicians, nurses, management, patients and their relatives, researchers, etc.). We already discussed the requirements of the various stakeholders in Sect. 1.3. If management of information systems is to be considered successful, then the information system should fulfill the requirements of these stakeholder groups. The strategic information management plan should thus be developed with input from major stakeholder groups within the health care facility. Details on how to develop a strategic information management plan were explained in Sect. 4.3.1.2.

This also means the IT project portfolio (Sect. 4.3.1) should be effectively managed in a way that maximizes the intended value to the health care facility and to the stakeholders. For example, projects that deliver the most business value (e.g., the introduction of a picture archiving and communication system (PACS) to support faster and better diagnosis and treatment) may be prioritized among other projects (e.g., a website for clinical staff informing on most recent business news) depending on the business goals.

Strategic monitoring should be done based on clearly defined key performance indicators (KPIs) and use data from several sources (e.g., user surveys, analysis of hotline calls). Evaluation projects should be conducted to assess the quality of certain parts of the information system. For example, after the introduction of a new computerized provider order entry (CPOE) system, changes in medication errors should be analyzed systematically. Details on how to plan and conduct evaluation projects are discussed in Sect. 5.4.

IT risk management should continuously assess risks and liabilities of information management. This includes the continuous identification, assessment, mitigation, monitoring, and management of risks related to IT. For example, an IT risk analysis may show that the whole hospital operation depends on the functioning of the communication server and of the IT network, and thus activities will be started to reduce the risk of network failures by increasing redundancy of technical components.

Finally, the architecture of the information system should be documented in an up-to-date way. It is surprising how many health care facilities do not have a consistent and clear description of their application components, the functions they support, and the interfaces between them. Establishing and using such an architectural description is an important activity within strategic information management planning. Using the three-layer graph-based metamodel (3LGM2) been described in Sect. 2.14 is helpful.

5.2.3 Quality of Tactical Management of Information Systems

Tactical management of information systems deals with particular functions, application components, or physical data processing systems. It aims to introduce, remove, change, or maintain components of the information system.

There are some ways to assess the quality of tactical management of information systems:

Projects of tactical management of information systems should be derived from the annual project portfolio. They should be conducted using best practices and state-of-the-art methods in project management in all project phases: project initiation, project planning, project execution, and project completion (Sect. 4.4).

All projects should be finalized within the planned time frame and within the planned budget. Changes in time frame and budget must be justified and approved by strategic management of information systems. Finally, and most importantly, projects should be successful, i.e., they should reach the intended project goals.

As part of tactical management of information systems, certification may also play a role. Certification in general means confirming that an object or organization has certain characteristics. Certification of health information systems in general describes a process where an accredited body confirms that the information system of a health care facility fulfills certain quality characteristics that have been predefined by an external organization. Examples of certification are provided in Example 5.5.3. Vendors may try to obtain these certificates for their application software products to obtain an advantage in the market. Health care facilities may check for the availability of these certificates when buying software for a new application component. In general, certification increases transparency of different products and fosters buyers’ knowledge about products, as certification organizations often compile information about the different products and technologies. Increased transparency and knowledge in turn have a positive impact on the buyers’ willingness to invest in new technology. Even when a certification does not guarantee that a component is good regarding all and every criterion, certification may contribute to an increased transparency of quality of health information systems in general.

5.2.4 Quality of Operational Management of Information Systems

Operational management of information systems is responsible for operating the components of the information system. It ensures its smooth operation in accordance with the strategic information management plan of the health care facility.

To assess the quality of operational management of information systems, we can assess several aspects.

First, we can assess whether the operation of the components of the information system (such as server, clients, networks, interfaces) runs smoothly and without frequent or longer interruption.

To minimize risks, a business continuity plan should be available that describes how the information system can continue to support the functions even in a case of larger failures or disasters (e.g., when the central server room of a health care facility is destroyed).

As part of IT governance, there should be a clear responsibility with documented tasks and processes for operating the physical data processing systems and the application systems in a health care facility.

Operational management of information systems should be based on established best practice frameworks and standards such as COBIT [1] as a framework for IT governance (Sect. 5.2.1) or Information Technology Infrastructure Library (ITIL) [2] for ITSM (Sect. 4.5).

ITSM provides a perspective on management of information systems that focuses on how IT is provided to serve the needs of customers. We can consider the management of IT services as that part of operational management of information systems focusing on responding to the customer’s needs. The term “IT service” in a health facility comprises the application systems (e.g., LIS or PACS) and the physical data processing tools (e.g., ward computers, mobile computers) discussed in Sects. 3.4 and 3.10. However, it also comprises more specific IT services such as remote VPN (virtual private network) access to the network of the health care facility, specific application interfaces, antivirus shielding, or videoconferencing facilities. All these are IT services that are needed by customers (e.g., by the clinical departments or the administration of a health care facility).

The desired content and quality of IT services should be clearly defined in service-level agreements (SLAs). An SLA describes, among other things, the processes supported by the IT service (e.g., access to images via a PACS), the desired outcome for the customer (e.g., mobile access to all patient images is possible for a physician), planned service times (e.g., 24/7), availability target (e.g., 99% availability of image access), allowed downtimes (only during night), number of users (e.g., 350 physicians), required levels of support (e.g., on-site support during day, remote support during night), and responsibilities (including duties of the service provider, duties of the customer, duties of the service user). An SLA is a contract regarding the provision of an IT service, for example, between a department and the IT department.

An IT service desk, also called helpdesk, should be available where users can report any incidents related to IT services and get help. The quality of the IT services should be continuously monitored and improved. All incidents related to IT services should be systematically documented, their root causes should be analyzed, and ways to prevent future problems should be developed and implemented.

Finally, IT staff should be sufficiently trained and competent to operate physical data processing systems and the computer-based applications.

5.3 Quality of Architectures and Infrastructures

In the previous section, we discussed the quality of the management of information systems. The outcome of high-quality information management is, hopefully, a high-quality information system. We will now discuss how we can assess the quality of the information system.

5.3.1 Quality at the Domain Layer

The overarching objective of a health information system is to support the functions of a health care facility. In Sect. 3.3, we presented these functions in more detail, such as patient identification, decision-making and patient information, execution of diagnostic, therapeutic, and nursing procedures, or billing.

Health information systems should sufficiently support information and knowledge logistics (Sect. 2.7) within patient care, administration, and management. To achieve a high quality of the information and knowledge logistics at the domain layer, information logistics should be as good as possible given the resources used. Good information and knowledge logistics comprises the following aspects:

  • The right information: Is the information that the users need available? For example, can the physician access the lab results of a patient?

  • At the right time: Is the information available when it is needed (just-in-time)? For example, are the most recent lab results available before the physician’s round starts?

  • At the right place: Is the information available where it is needed? For example, are the recent lab results available at the patient’s bedside? Is information also available across health care facilities to support continuity of care (e.g., can hospital, nursing home, and general practitioner (GP) access relevant information on a given patient treated by all three facilities?)

  • To the right people: Is the information available only to those who need it and who have the right to see the information? For example, are the recent lab results only available to the treating health care professionals? Is data protection guaranteed? Is relevant information available not only for health professionals, but also for the patient and relatives?

  • In the right form: Is the information available in a usable format? For example, can lab results also be displayed in a graphical way for a longer period of time? Can information be personally filtered (personal filtering), not overwhelming the health care professional with too much information (information overload)?

  • In order to make the right decisions: Information should be used to inform decisions. For example, a graphical presentation of the lab results may show an increase in a lab value, which may in turn motivate the physician to change the administrated drug.

In addition, the domain layer describes the data to be processed and provided. To assess the quality of the data at the domain layer, we can assess the following data quality aspects:

  • Data integrity should be maintained. Data integrity means that data are consistent, that object identity is maintained, and that relationships between entities are correct (referential integrity) (Sect. 3.5). For example, every patient and every patient case should have a unique PIN that is used in all application components.

  • Data in general should also be correct and have an indisputable authorship. For example, the author of every patient report should be clear and verifiable. Clinical data should also be kept confidential. For example, the diagnoses of a patient should only be accessible to the treating health care professionals.

  • Finally, data should be uniformly recorded. There should be clear rules on which data and how the data are recorded and stored. Standardized data can be better processed automatically, while non-standardized data can provide more specific information to a human reader (Sect. 3.2.2).

5.3.2 Quality at the Logical Tool Layer

The information system quality at the logical tool layer comprises the quality of application components and the quality of their integration.

The following criteria help to assess the quality of an application component:

  • Application systems should offer the features required for a given process. For example, a radiology information system (RIS) should offer features required for report writing, it should provide correct output when searching for a patient, and it should guarantee security of stored data.

  • As clinical workflows often change, an application component should be adaptable to workflow changes.

  • An application system should be reliable and provide defined services for a defined time under the given conditions. For example, an RIS should have little or no downtimes. An application system should also be easy to maintain. For example, an upgrade of the RIS software must be done quickly and without endangering the overall application component’s stability.

  • An application system should be user-friendly. Good usability is very important for software used within health care facilities. Health care professionals spend only a small amount of their working time with computer-based tools, and they often have to use various application components for their work. In addition, staff turnover is very high. Therefore, application software products should be easy to learn and intuitive to use. This is addressed by ISO 9241-171, which defines specific quality characteristics for software ergonomics, such as self-descriptiveness or tolerance with regard to user errors.

  • An application system may need to be certified depending on the legal regulations. For example, the PACS software is certified according to the national law for medical devices.

  • Finally, an application system should offer standardized interoperability interfaces to facilitate integration with other application components (Sect. 3.7.2). For example, an RIS may offer a Health Level 7-based (HL7-based) interface for data exchange with the patient administration system or with other clinical systems. Otherwise, integration in ACn architectures is hardly reachable in an economically reasonable way, as proprietary interfaces between two application components are expensive to develop and to maintain.

The general architecture of the health information system should be sufficiently flexible to adapt to the changing needs of the hospital. For example, it should be easy to add new application systems to the information system, and application components should be easily replaceable by other (more advanced) application components. A star-based architecture (CP1 architecture) with a communication server (Sect. 3.9.2) and application components offering standardized interfaces support the exchange or addition of application systems.

An architecture can be called “saturated” if as many functions as possible are supported by computer-based tools and if there are no or only a small number of non-computer-based tools still in use. Please note that computer-support is not a goal in itself. However, a mix of paper-based and computer-based tools within a function often leads to deficiencies in information logistics such as transcriptions. Transcription means the manual transfer of data from one application component to another, for example, manually entering patient diagnoses from the patient administration system in the CPOE system or scanning a discharge letter to add it to the electronic patient record (EPR). Transcriptions are time-consuming and may lead to data errors. Therefore, transcriptions have to be avoided by using standardized interfaces between application components.

To achieve integration in “best-of-breed” architectures (ACn, Vn), application components need to share and store the same data. Data redundancy is thus unavoidable. For example, patient administrative data are stored in the patient administration system, the RIS, and the LIS. This data redundancy needs to be closely managed to provide consistent data. Approaches for handling data redundancy through integration technologies were presented in Sect. 3.9.

5.3.3 Quality at the Physical Tool Layer

The information system quality at the physical tool layer comprises the quality of physical data processing systems and their integration.

The quality of physical data processing systems can be described by several characteristics:

  • Physical data processing systems should be available where needed (e.g., at the patient bedside). They should be stable and reliable, i.e., without unexpected downtimes. They should be performant to allow fast processing and the ability to present large amount of data (including images).

  • Physical data processing systems should be secure, for example, following rules for data safety, data security, and electrical safety. They should be user-friendly (e.g., allowing data entry via touchscreen, mouse, and keyboard). In certain areas, they need be certified. For example, the tools used in the intensive care unit (ICU) need to be certified according to the national law for medical devices.

  • Physical data processing systems should be usable for several tasks. For example, a mobile tool (such as a notebook) on a ward should allow access to the nursing documentation system, the CPOE system, and the patient chart. This limits the risk that users have to handle multiple physical tools at the same time and thus supports the “leanness” of information-processing tools.

At the physical tool layer, redundancy is often valuable to reduce risks of system failure. For example, data may be stored redundantly in different areas in order to avoid data loss in case of fire. Or data may be duplicated on different hard discs in a specific database server (e.g., using redundant array of independent discs (RAID) technology), allowing reconstruction of data when a hard disc fails. Thus, technical redundancy is also an important quality criterion for the physical tool layer.

5.3.4 Quality of Integration

In Chap. 2, we learned that health information systems are constructs built from a variety of components. In Sect. 3.7, we discussed that these components, especially application systems, need to be interoperable so they can be integrated to best support functions and business processes. Integrating application systems, as we saw in Sect. 3.8, can be done in a number of ways, each achieving specific qualities of the information system. We will therefore summarize these types of integration here again as quality criteria for health information systems.

Data integration is achieved in a health information system when data that have been recorded in different application components once are available wherever they are needed without having to be reentered (Sect. 3.8.1). Consequently, in a health information system where data integration is given, data can be brought together for analysis wherever it is needed. Moreover, if the data needs to be updated, this only has to be done in one place, even if the data are redundantly stored in several application systems. Overall, data integration is the first and quite basic quality characteristic within heterogeneous health information systems, as it allows application systems to exchange and reuse data while preserving data integrity.

Semantic integration (Sect. 3.8.2) is guaranteed if different application systems use the same system of concepts, i.e., they interpret data the same way. Semantic integration is an important quality characteristic within heterogeneous health information systems, as it supports the exchange of meaningful information between application systems.

User interface integration (Sect. 3.8.2) is guaranteed when different application components represent data and organize their user interfaces in a unified way. User interface integration supports the usability of application systems and reduces errors when searching for or entering data. It thus also contributes to data quality and patient safety, making it an important quality characteristic within heterogeneous health information systems, as it supports ease-of-use and reduces usage errors of graphical user interfaces.

Context integration (Sect. 3.8.4) is an important quality characteristic within heterogeneous health information systems, as it allows synchronizing and coordinating context among application systems. It thus allows application systems to automatically follow patient, user, and other contexts and thus supports the user when working with several application systems. Note that context integration stands on its own. It neither contributes to data, to semantic, or to user interface integration. Vice versa, these types of integration will not support achieving context integration.

Feature integration (Sect. 3.8.5) means that features are not implemented redundantly in multiple application systems. Feature integration thus reduces costs for both implementation and maintenance of application systems. Overall, feature integration is an important quality characteristic within heterogeneous health information systems, as it allows sharing of functions among application systems.

An integrated health information system should support the business processes effectively. From this perspective, process integration (Sect. 3.8.6) is indeed the overall vision of integration within heterogeneous information systems. Process integration is guaranteed when business processes are effectively supported by a set of interacting application systems. Systematic adoption of Integrating the Health care Enterprise (IHE) profiles is an indicator for structural quality on which smooth process integration can be achieved. Process integration is an important quality characteristic within heterogeneous health information systems, as it describes a situation where different application systems interoperate in an optimal way so that business processes are best supported.

5.4 Evaluating the Quality of Health Information Systems

Evaluation can be defined as the act of measuring or exploring components of a health information system. The result of an evaluation should provide information to support decisions concerning the health information systems, such as decisions regarding optimizing, replacing, or further deploying a component.

This definition of evaluation highlights the fact that evaluation can comprise both quantitative (“measuring”) as well as qualitative (“exploring”) aspects and that evaluation should answer a clear question and thus support management decisions of strategic or tactical management of information systems. Evaluation studies can, for example, help to justify IT investments, to verify that the information system is effective and safe, or to understand problems and to improve the information system.

We will now discuss the basic phases of an evaluation study. We will see how to identify an evaluation question together with stakeholders, how to collect quantitative and qualitative data, and finally how to answer the evaluation question and how to use the evaluation results to improve the health information system. We will provide only a short introduction to this topic. Please consult specific textbooks to learn more about health IT evaluation (such as [3]).

5.4.1 Identifying the Evaluation Question

Evaluation studies of components of health information systems should answer a clear question and thus inform a decision. Such a decision may focus on ways to improve a component or the need to replace a component by another one. Identifying an evaluation question that is useful in a given situation is thus the first crucial step for any evaluation.

Which evaluation question is useful depends on the context and especially on the adoption phase of the component. Adoption can be described as the successful integration of an innovation in a health care facility. Adoption is a time-dependent process.

The Clinical Adoption Metamodel [4] describes four dimensions of adoption of application systems that depend on each other: The first dimension of adoption is “availability,” which comprises the ability of users to access the system, the availability of the system, and the availability of content and features of the system. The second dimension of adoption is “system use,” comprising the actual use of the system and the subjective user experience with the system. The third dimension of adoption focuses on “clinical behaviors” and comprises the meaningful adaptation of clinical processes to the system. The fourth dimension of adoption is reached when the new system has an impact on “clinical outcome” at the patient level, the provider level, or the population level. Combined, these four dimensions describe an adoption trajectory from first implementation of a new application system (or a specific feature of it) to changes in outcomes.

This adoption model is helpful to identify the evaluation questions that are most relevant for given situation. In other words: Depending on the state of adoption of an application system, only specific evaluation questions are of relevance and make sense. For example, imagine that a health care facility has introduced a CPOE system for medication ordering with the aim to increase efficiency and quality of prescriptions.

  • In the adoption phase of “availability,” evaluation may focus on the following question: Is the CPOE system sufficiently made available to the intended user groups, for example, do all relevant users have a user account and are sufficient mobile tools for prescriptions available on all wards? Is the system available as planned, for example, are there no unplanned downtimes and is performance and stability as planned? Is all needed information available within the CPOE system, including patient administration data, prescription information, drug information databases, and interaction checks? Are all interfaces working as planned? Are the users sufficiently trained on the CPOE system, for example, have all physicians and nurses received sufficient training and support?

  • In the adoption phase of “system use,” evaluation may cover the following questions: Is the CPOE system being used as intended by the various user groups, for example, are all prescriptions entered directly by the physicians into the CPOE system during ward rounds? Are all main features of the CPOE system being used as intended, for example, are interaction checks used at all? Do the users consider the system to be user-friendly?

  • In the adoption phase of “clinical behavior,” evaluation will focus on the processes: Are the prescription processes efficiently supported by the system? How many automatic alerts of interaction checks occur, and are they considered and reacted upon by the clinical users? Did the overall number of prescriptions change after introduction of the CPOE system?

  • In the adoption phase of “clinical outcome,” evaluation will assess improvement in patient outcomes (e.g., reduction of medication errors, increase in patient safety) or costs (e.g., reduction of medication costs).

Evaluation questions thus have to be properly chosen depending on the adoption phase of an application component and depending on the decision that is to be supported by the evaluation, such as decision on further rollout, on further upgrades or other technical improvements before rollout, or on cancellation of a pilot project.

Given the usual constraints of time and money, an evaluation should focus on a limited set of evaluation questions and not try to answer too many questions. To increase the chance that evaluation focuses on an evaluation question that is useful to decision-makers, the decision-maker needs to be consulted when defining the evaluation question.

Identification of the most relevant evaluation questions can be done, for example, in a workshop with the decision-maker, where the following questions could be discussed:

  • What is the object that should be evaluated, i.e., which application system or which feature should be evaluated?

  • What is the phase of adoption the application system is in at the moment?

  • Which decision are to be made regarding the application system and how can evaluation results help in this decision?

  • Which evaluation questions would provide the most crucial information?

Such a workshop between the CIO, the medical director, and the evaluator, for example, could show that the CPOE system is already well adopted in one pilot department. It is now important to better understand the effect of the CPOE system on medication errors and patient outcome to be able to decide on further rollout. The major evaluation question will therefore be: “Does the CPOE system improve medication safety?”

After defining the evaluation questions, clear indicators need to be defined. For example, medication safety may be measured by counting prescription errors or adverse drug events or by looking at length of stay or mortality. Which indicator is best to answer a given evaluation question needs to be carefully decided based on the context, the available data, and the available scientific literature.

5.4.2 Deciding on the Study Design

Depending on the study question and considering the context of the study (e.g., available resources), the study design needs to be carefully chosen. Several types of study designs exist:

First, we have to decide whether the study is planned as a quantitative study, a qualitative study, or a mixed-method study. Quantitative studies focus on collecting quantitative data (e.g., number of patient safety incidents or number of user logins) to answer the study question, while qualitative studies collect qualitative data (e.g., free text comments within surveys or user comments from interviews). Mixed-method studies combine quantitative and qualitative data.

Second, we have to decide whether to conduct an exploratory study, a descriptive study, or an explanatory study. Explorative studies try to explore and describe a given situation, which means generating information to improve understanding of the situation. For example, an explorative study may try to find out the reasons for higher medication errors after introduction of a CPOE system. Explorative studies are typically qualitative or mixed-method studies (i.e., they often apply open-ended observations or interviews). Descriptive studies focus on measuring a predefined attribute in a group of objects. For example, a descriptive study may measure user acceptance of CPOE systems or IT knowledge of the hospital staff. Descriptive studies are typically quantitative studies (i.e., they often apply a standardized survey or standardized observations). Finally, explanatory studies try to assess predefined hypotheses. For example, an explanatory study may test the hypothesis that introducing a CPOE system significantly reduces medication errors in a given clinical setting. Explanatory studies are typically quantitative studies (i.e., they use quantitative measures to determine changes of the effect).

Third, in case an explanatory study is planned, we have to decide whether to conduct it as an experimental study, a quasi-experimental study, or a non-experimental study. For experimental trials, the randomized controlled trial (RCT) is considered the gold standard with highest internal validity. By conducting an RCT, we can determine with a certain probability whether a given intervention (e.g., a CPOE system) led to a certain effect (e.g., a reduction of medication errors). Quasi-experimental study designs also try to assess a relationship between an intervention and an effect but have less internal validity compared to RCTs. Quasi-experimental designs are, for example, before-after trials or trials with a non-randomized control group. Both experimental trials and quasi-experimental trials are also called intervention studies, as they comprise a predefined intervention. Non-experimental studies (also called observational studies) also try to assess the relationship between an intervention and effect, but they do not intervene in any way, instead observing and collecting available (quantitative) data. They can be performed as cross-sectional studies (i.e., collecting data only at one point in time) or as longitudinal studies (i.e., collecting data at several points in time, for example, every 3 months).

Fourth, we have to decide whether to conduct a laboratory study or a field study. In laboratory studies, we can better control the overall study setting, yet the external validity, i.e., the transferability of results to real settings, is limited. In field studies, it is more difficult to control the overall study settings, but external validity is higher.

5.4.3 Collecting Quantitative Data

Quantitative data comprise data that can be described by numbers. Numbers can easily be statistically analyzed, aggregated, and compared. The basic idea of quantitative evaluation methods is that objects have attributes (such as duration or amount) that can be exactly measured. To obtain data representative for a predefined population, a sampling is selected and then analyzed.

Typical quantitative evaluation methods comprise time measurements, quantitative observations, or quantitative surveys.

Time measurements comprise time-motion analyses or work-sampling studies. The time-motion analysis is based on trained observers that measure the duration of observed events (e.g., tasks) while using a predefined list of event categories. Typically, for time-motion analysis, one observer is needed for each observed actor (e.g., user), which is quite resource-intensive. This disadvantage is resolved by the work sampling analysis. Here, trained observers document which task is being executed only at predefined (e.g., every 5 min) or randomly selected time intervals. This allows them to observe several actors in parallel. By counting the number of observed tasks in each category, the overall distribution and thus the duration of each task can be calculated. To obtain precise numbers, however, this requires a relatively large number of observations to be done. Please note that both approaches (time-motion and work sampling) are typically conducted by trained external observers, though they can in principle also be conducted by the actors (e.g., users) themselves—this, however, may limit the quality of the data.

Quantitative observations comprise observations of clinical situations or processes or the analysis of available data (e.g., log files) in which the number of events that occur in a given time period is counted by an observer. This can, for example, involve counting medication errors (based on an analysis of patient records), counting the number of clicks when using certain software, counting the number of physician–patient interactions, or counting the number of patients entering a department. As for any measurement, special attention should be given to training the observers and to using standardized observation protocols to achieve inter-observer reliability.

Quantitative surveys use standardized, closed questions that lead to quantitative results. For questionnaires addressing subjective opinions and feelings, the 5-point Likert scale is often used (“strongly agree”—“agree”—“neither agree nor disagree”—“disagree”—“strongly disagree”). The quality of data achieved by questionnaires depends on a thorough formulation of questions and predefined answer categories. Each questionnaire should be pretested. The available literature should thus be consulted before planning a questionnaire in order to ensure objectivity, reliability, and validity of results. If possible, available and validated questionnaires should be reused.

5.4.4 Collecting Qualitative Data

Qualitative data comprises text and any other non-numerical data. Qualitative data can describe individual situations and contexts in quite some detail. Typical qualitative evaluation methods comprise qualitative interviews, qualitative observations, or qualitative content analysis.

Qualitative interviews comprise all forms of semi- or unstructured interviews that use open questions, thus generating free text as a result. This allows the respondent to answer freely and allows interaction between interviewer and respondent. The interview can be conducted with one or more respondent at the same time. Group interviews support interaction between respondents but should only be done in groups without hierarchical dependencies. In any case, a pretested interview instruction is needed that describes how the interviewer should conduct the interview and document the results. Answers are typically recorded by tape and later transcribed in verbatim or aggregated protocols. The data can be analyzed using qualitative content analysis, for example.

Qualitative observations comprise open, less-standardized, non-quantitative observations of processes or events. Contrary to quantitative observations, the aim is not to count and measure, but to obtain deeper insight into a situation. The observations are typically documented in a field diary or on predefined observation protocols. Qualitative observations generate text (such as observer notes) that can be analyzed using qualitative content analysis. Please note that for qualitative observation, there should be a certain familiarity of the observer with the observed field (e.g., with the situation in the clinical department).

Qualitative content analysis is used to analyze text that obtained from qualitative interviews or qualitative observations. Qualitative content analysis is a methodological, planned approach for grouping qualitative data into categories. Here, the material is analyzed stepwise and coded into several available categories. The categories can either be defined beforehand (deductive approach), developed while reading and analyzing the text (inductive approach), or defined beforehand and refined while analyzing the text (mixed approach). The coding of text into categories should be reproducible; it must therefore be clearly documented and explained with the so-called anchor examples. Typically, the text material is read and coded more than once to make sure that nothing is overlooked and that the categories are homogeneous and all filled with text examples. Based on the categories and the text passages that are related to them, the text can then be further analyzed to identify larger patterns and to answer the study questions.

5.4.5 Answering the Evaluation Questions

If an evaluation was carefully planned, the collected quantitative and qualitative data should now allow the previously defined evaluation question to be answered. For example, a pre-post study of prescription errors showed that prescription errors were largely reduced by the CPOE system. This result motivates further rollouts.

Evaluation results may help to improve the quality of the information system, for example, the quality of integration (Sect. 5.3). Evaluation studies that aim mostly at collecting data to improve an information system are also called formative evaluation studies. Formative evaluation studies often take place in early phases of adoption. For example, a typical formative evaluation assesses whether users are using a CPOE system as intended by conducting qualitative observations to identify if there is need for further user training. Results of formative evaluation will thus help to decide on needs for improvement in technology, processes, or training.

Evaluation results may also answer the question whether the application system has achieved its intended goals. These evaluations typically focus on a certain outcome of a component and take place in later adoption phases. These types of evaluation studies are also called summative evaluation studies. For example, a typical summative evaluation assesses whether the CPOE system has improved medication safety after 1 year of using the CPOE system by applying quantitative chart analysis. The results of summative evaluation will help to justify the expenses of the CPOE system and motivate further rollout.

5.5 Examples

5.5.1 Unintended Effects of a Computerized Physician Order Entry Nearly Hard-Stop Alert

The introduction of application systems may have unintended effects. The careful evaluation of impact and unintended effects of application systems is thus an important task of management of information systems. We will now have a look at an example of an evaluation study that showed some unintended effects of CPOE systems.

Table 5.1 presents the abstract of an RCT on automatic alerts in a CPOE system. The authors analyzed whether the so-called hard-stop alert can reduce unwanted drug–drug interactions. Such a “hard-stop alerts” appears on the screen to alert the physician about potential problems associated with a particular prescription and blocks the clinician’s order from further execution to avert potentially serious reactions.

Table 5.1 Abstract from “Unintended Effects of a Computerized Physician Order Entry Nearly Hard-Stop Alert” [5]
Full size table

The study was designed as a quantitative, explanatory field study that was conducted as an RCT. The study found that these alerts can help to reduce the number of alert-triggering orders. But it also found that the hard-stop alert led to clinically important treatment delays in four patients.

5.5.2 Clinical Decision Support for Worker Health: A Five-Site Qualitative Needs Assessment in Primary Care Setting

Besides evaluating the effect of an intervention, evaluation may also try to understand reasons for successful or unsuccessful implementation of an application system. For these kinds of questions, qualitative studies are often chosen.

Table 5.2 presents the abstract of such a qualitative study. The authors analyzed need, barriers, and facilitators for clinical decision support (CDS) in primary care. The study was performed as a qualitative, exploratory field study.

Table 5.2 Abstract from “Clinical Decision Support for Worker Health: A Five-Site Qualitative Needs Assessment in Primary Care Settings.” [6]
Full size table

The authors found several factors that may hinder or foster the use of CDS in primary care. The results of this multi-center study can now be used to implement CDS in commercial application software products for primary care.

5.5.3 Certification of Health Information Systems

There exist several national and international approaches for certification of application software products to be used in health care, such as the European CE certification, the German digital health application (DiGA) repository, the ONC Health IT Certification Program in the U.S. and IHE Connectathons.

CE certification is mandatory in the European Union for all application software products developed to be used for medical purposes such as diagnosis, prevention, monitoring, or prognosis. For example, a software calculating the correct amount of insulin needed is considered a medical device and thus needs CE certification. CE certification assures that the software complies with all European legal requirements regarding safety, health, and environment. Depending on the risk class that the application software product belongs to, certification can be done by the manufacturer of the software or it must be done through external auditing. Details on CE certification are available in the EU’s Medical Device Regulation 2017/745 [7].

The German DiGA repository for “health applications” [8] is maintained by the Federal Institute for Drugs and Medical Devices and lists health apps that fulfill a set of quality requirements. Besides being CE-certified as a medical device of a low-risk class, the application must be actively used both by the patient and by a health care provider and must fulfill certain data protection and information safety requirements. In addition, the vendor must provide supporting evidence (e.g., from evaluation studies) about the positive effect of the health application on the quality of patient care. When listed in the DiGa repository, health applications can be prescribed by a physician and are reimbursed by the patient’s health insurance company.

In the United States, the Health IT Certification Program is operated by the Office of the National Coordinator for Health Information Technology (ONC) [9]. To be certified, a vendor of a component must fulfill a number of requirements that assess whether an application software product supports clinical processes, care coordination, privacy and security, patient engagement, and exchange and interoperability of patient data. Part of certification is the annual testing of the component in real-world settings.

The IHE initiative [10] (Sects. 3.7.2.5 and 3.7.2.6) strives to increase interoperability between components based on existing standards such as HL7 and DICOM (Digital Imaging and Communications in Medicine). IHE offers testing for the standards-based interoperability between components in the so-called Connectathons. During a Connectathon, components of different vendors exchange information with other components in a supervised environment. The Connectathon provides detailed validation of the components’ interoperability and compliance with IHE profiles. The results of testing are published by IHE.

Besides certification of quality and interoperability of software, health care facilities or vendors can strive for certification of the quality of their internal processes by applying for ISO certifications or for the Joint Commission certification.

The ISO 9001 standard [11] defines criteria for the quality management system of an organization. An ISO 9001 certificate states that an organization follows certain formalized quality processes, that it monitors the outcome of its processes, and that it facilitates their continuous improvement.

The ISO 27001 standard [12] focuses on information security management. An ISO 27001 certificate states that an organization has an established information security management system and is able to manage the security of information, such as employee information, patient information, or financial information.

The Joint Commission [13] is a US-based organization assessing health care organizations and programs based on preestablished quality standards. A subset of its standards focuses on management of information systems and addresses issues such as data privacy, information security, standardization of data collection, and interoperability of data.

5.6 Exercises

5.6.1 Quality of Integration

Read the following case descriptions and discuss the integration problems using the types of integration presented in Sect. 5.3.4. Which negative effects for information logistics result from the identified integration problems?

  1. 1.

    A physician enters a medical diagnosis for a patient first in the medical documentation and management system (MDMS) and later, when ordering an X-ray, again in the CPOE system.

  2. 2.

    The position of the patient’s name and the formatting of the patient’s birthdate vary between the MDMS and the CPOE system.

  3. 3.

    When physicians shift from the MDMS to the CPOE system, they have to log in again and again search for the correct patient.

  4. 4.

    The CPOE system and the RIS use slightly different catalogs of available radiology examinations.

  5. 5.

    When physicians write the discharge letter for a patient in the MDMS, they also have to code the discharge diagnosis of a patient. For this coding, they have to use a feature that is only available in the patient administration system, so they have to shift to this application system.

  6. 6.

    While at the patient’s bedside during their ward rounds, physicians have to use several application components at the same time, such as MDMS for retrieving recent findings, the CPOE system for ordering, and the PACS for retrieving images.

5.6.2 Data Collection in Evaluation Studies

Read Examples 5.5.1 and 5.5.2 and determine which methods for collecting data (as described in Sects. 5.4.3 and 5.4.4) have been used.

5.6.3 Study Design in Evaluation Studies

Read Examples 5.5.1 and 5.5.2 and describe the chosen study design in more detail, using the description presented in Sect. 5.4.2.

Try to explain for which types of study questions the RCT is the best study design.