1 Introduction

Emergency Department (ED) patient presentations can cover a wide range of symptoms and conditions requiring diagnosis, management, and/or treatment. The use of diagnostic tests, including laboratory tests, can provide clinicians with information to aid diagnosis and help inform medical decision making. Laboratory tests are a critical part of the ED process. This is evidenced by a survey of ED physicians from a Netherlands hospital which rated clinical chemistry analyses as the third most important diagnostic tool (Janssens et al. 2014) and a US study reporting that a minimum of one laboratory test was ordered in 56% of ED presentations (Ngo et al. 2017).

Despite the importance of diagnostic testing in health care, there is evidence of considerable variation in diagnostic testing across hospitals (DePorre et al. 2020), including hospital EDs (Florin et al. 2013; Wabe et al. 2019). Diagnostic under-testing can lead to patient safety concerns associated with missed or delayed diagnosis (Medford-Davis et al. 2016), and over-testing can lead to added investigations resulting in unnecessary procedures and/or potential risk to patients (Hofmann and Welch 2017; Gion et al. 2022). Over-testing can also increase the costs associated with healthcare provision (Tamburrano et al. 2020; Frutos et al. 2022) and can have environmental impacts (Breth-Petersen et al. 2022). ‘Reducing low-value care’ was the focus of the Australian Choosing Wisely 2021 report which included case studies of successful initiatives undertaken in Australian hospitals to reduce unnecessary test ordering (NPS MedicineWise 2021).

Digital health applications used in healthcare settings frequently include functionality to facilitate or moderate electronic ordering of laboratory tests including, for example, clinical decision support systems (CDSSs) (Hughes and Jackups 2022). Digital health can contribute to effective decision-making by improving the ability to gather, organise and display information, or by enhancing timely access to diagnostic reference information, facilitating follow-up, and feedback to clinicians and patients (El-Kareh et al. 2013). Systematic reviews have identified the most common characteristics of CDSSs (Hak et al. 2022) and the wide variety of CDSS types/approaches (Patterson et al. 2019; Hak et al. 2022), including CDSSs for laboratory test ordering (Zare et al. 2021) and electronic health record-integrated CDSS implemented in ED settings (Patterson et al. 2019). Although reviews have reported positive outcomes from CDSSs (Patterson et al. 2019; Zare et al. 2021), there is also evidence of low uptake/use of CDSSs in the clinical setting (Patterson et al. 2019; Kouri et al. 2022; Salwei et al. 2022). Studies have highlighted the need for greater emphasis on people/human centered design (Patterson et al. 2019; Noteboom et al. 2022), processes/clinical workflows (Noteboom et al. 2022; Salwei et al. 2022) and clinician involvement in CDSS design (Khairat et al. 2018) to improve usability and acceptance.

Our approach to the design and development of the Emergency Department Pathology Order Support Tool (ED POST) prototype draws on lessons-learned and recommendations from the literature. The current protocol details the methodological approach to be undertaken during stage 1 (pre-prototype development) and stage 2 (protype evaluation), which will be qualitative, workflow focused, and involve close collaboration with ED clinicians. The development stage (prototype build) will involve the technical design and development of the prototype which will be briefly described herein, with the technical specifications reported in greater detail post-development.

2 Conceptual overview

The ED POST concept is founded on the need to address unwanted variation in laboratory test ordering to reduce the impacts of over/under testing. The goal of the project is to improve consistency in test ordering within the ED setting, through the design and development of an intelligent and dynamic electronic tool to facilitate appropriate and effective use of laboratory/pathology test ordering. Conceptually, ED POST proposes the utilisation of routinely collected data from the New South Wales (NSW) Health Pathology Atlas of Variation (detailed elsewhere in the literature (Scowen et al. 2020; Wabe et al. 2021a, 2021b)) which is contextually appropriate to the ED setting. ED POST is proposed to serve as a dynamic clinical decision support tool presenting information to ED clinicians (medical and nursing) based on data that is continually collected from pathology tests ordered across EDs in the Australian state of NSW. The prototyping aims to achieve a scalable digital health support tool that allows clinicians to enter patient variables at triage and return probability scores to assist with decision making on the pathology order, for almost every possible presenting scenario. Probability scores will be calculated using a standard Boosted Decision Tree Machine Learning Algorithm available within the Microsoft Azure Machine Learning platform. This process is to establish base score as a continuous variable for any combination of the input variables. Error checking, impact of small sample scores, evaluation of gender/diversity biases and clinical review will be built into the process beyond the prototype. ED POST has the potential to incorporate state-wide, Local Health District and facility scores, adjusting for seasonality and geographic nuance.

The research methods detailed in the current protocol will be used to collect qualitative data to inform the realisation of the ED POST concept into a prototype design. The ED POST concept of utilising routinely collected data (from the NSW Health Pathology Atlas of Variation) to inform pathology ordering in the ED setting will be discussed with clinicians to gain feedback and input to determine the functional requirements and design of a prototype.

The design and development of ED POST will be undertaken across three stages (Fig. 1). The current protocol will detail the qualitative methods to be undertaken during stage 1 (pre-development) and stage 2 (prototype evaluation) including the data collection and analysis methods.

Fig. 1
figure 1

A diagrammatic representation of the three stages of the ED POST design and development. Stage 1 includes the predevelopment activities involving qualitative research with Emergency Department clinicians. The data collection methods include semi-structured interviews, think aloud observations and input on the design concept (see Sect. 5.1). The data analysis outputs from Stage 1 will inform the initial prototype development. The development stage involves the technical development activities which will be reported separately post-development. Stage 2 constitutes qualitative research methods including a focus group and demonstration (see Sect. 5.2) which will be used to undertake the prototype evaluation (usability, utility and suggestions for improvement). The outcomes of stages 1 and 2 will inform the development stage. Stage 2 may be iterative if it is necessary to develop multiple versions of the prototype

Full size image

The development stage of ED POST, which involves the technical design and development of the prototype, will be informed by the outcomes of the pre-development and prototype evaluation stages.

3 Aims and objectives

The global aim of the current protocol is to detail how the qualitative methodology and associated methods will be applied during the pre-development (stage 1) and prototype evaluation (stage 2) stages of the ED POST prototype design and development. The aim of the research study is to identify and understand existing pathology test ordering processes in the Emergency Department to inform the design and evaluation of the ED POST prototype, and to conduct a clinician-focused evaluation of the prototype for further development. The scope of the current protocol is centred on the qualitative methodology to be used in the development and evaluation of a prototype design, which must be completed before any implementation/intervention-based studies can be planned or undertaken. The objectives are:

Stage 1 To inform the design and development of an ED POST prototype based on ED clinician needs, through understanding existing work processes and decision-making requirements for pathology test ordering in the ED; and to determine clinician perspectives of potential barriers/challenges to the design and development of the proposed tool.

Stage 2 To conduct an evaluation of the ED POST prototype to gain clinician feedback on the design, usability, and utility of the tool, including suggestions for improvement; and to determine any impact on work processes.

The planned outcomes of the current research will be:

Stage 1 (i) an understanding of the workflows and decision-making requirements for pathology test ordering in the ED. Business process workflow process maps will visually document these processes. (ii) Understanding perceived barriers and challenges for consideration in designing ED POST from the clinicians’ perspective. Stage 1 outcomes will inform the development stage.

Stage 2 An evaluated prototype with clinician feedback documented to inform further/revised development. The outcomes of Stage 2 will inform further modifications during the development stages.

4 Study design

The ED POST design and development study will have a qualitative exploratory design (Rendle et al. 2019) aimed at understanding processes and workflows and gaining comprehensive clinician input and feedback prior to, and after, the development of the prototype. A hypothesis is not being tested, rather, the study will explore work practices to inform the design and development of a computerised decision support tool.

The research will draw on the elements and attributes of existing design methodologies in the literature including (i) design thinking (Roberts et al. 2016; Micheli et al. 2019; Krolikowski et al. 2022) which has been described as “…a systematic innovation process that prioritises deep empathy for end-user desires, needs and challenges to fully understand a problem in hopes of developing more comprehensive and effective solutions” (Roberts et al. 2016 p. 12); and (ii) human-centred design in which human needs are at the core of the design process (Maguire 2001; Melles et al. 2021). Furthermore, the qualitative methodology and associated methods to be used in the design and development of ED POST are cognisant of peer lessons-learned and recommendations detailed in the literature. The methodology will thus be founded on (i) close collaboration with ED clinicians (medical and nursing) (Maguire 2001; Micheli et al. 2019; Krolikowski et al. 2022) from the outset of project (prior to development of a prototype) and continuing throughout prototype development and evaluation (Melles et al. 2021); (ii) gaining an in-depth understanding of the context and requirements of ED pathology ordering (Maguire 2001; Roberts et al. 2016; Micheli et al. 2019; Melles et al. 2021) and (iii) involving ED clinicians in the evaluation of the initial prototype to inform refined development (Maguire 2001).

A qualitative research approach (Busetto et al. 2020) has been chosen as the most in-depth way of understanding how pathology test ordering occurs in the ED and the decision-making processes used in determining the types of tests to be ordered. Qualitative methods will also allow designers and researchers to gather suggestions and feedback from clinicians on the proposed design of ED POST including demonstrating/visualising the prototype using a focus group approach to inform refined development. Qualitative data will also be used to inform the development of business process maps to visualise (Rojo et al. 2008; Micheli et al. 2019) workflows and determine the impact of the proposed intervention on existing processes.

4.1 Setting and participants

The ED POST qualitative research will be undertaken in an Australian public hospital. In the Australian healthcare system, patients can self-present to an Emergency Department at any time, without requiring a doctor’s referral. Emergency departments also receive patients arriving via ambulance. The study site for the pre-development phase will be a metropolitan teaching hospital with a 24 h Emergency Department, which averaged over 80,000 annual attendances between 2018 and 2022 (Bureau of Health Information 2023).

Clinicians (medical and nursing) who have responsibility for pathology test ordering in the ED will be invited to participate in the study. Participation will be sought from junior clinical staff through to senior clinicians and department managers. Participation, with informed consent, will be entirely voluntary and no financial incentives/reimbursements will be offered. Participants involved in stage 1 of the study will be invited to continue to participate in stage 2 of the study, with additional recruitment as necessary. During the prototype evaluation phase, Emergency Department clinicians from additional public hospitals will be invited to attend the prototype demonstration as external stakeholders. The additional recruitment will enable the research team to evaluate the extent to which the prototype is scalable beyond a single site. Designing a scalable solution will be an important focus across all phases of the design and development of the ED prototype.

4.2 Sample size

As there is no one single method for determining a-priori sample size in qualitative research, the sample size for the current study has been estimated and guided by the use of the “Define, Explain, Justify, Apply” (DEJA) tool as detailed by Mthuli et al (2021). Define: the study will use non-probability sampling i.e. non-random sampling (Etikan et al. 2016), as the study objective requires specific knowledge in the area of pathology ordering processes. Explain: purposive expert sampling (Etikan et al. 2016) will be used to recruit participants who have expert knowledge and an in-depth understanding of pathology test ordering and decision-making processes in the ED setting. Justify: A purposive sampling approach will ensure participants have both experience in pathology test ordering and experience working in the ED. This is necessary as the intervention is being developed for use in a specified environment (the ED) and the evaluation involves a decision support prototype customised specifically for that environment. Apply: As the study will initially commence at a single hospital ED, a sample size estimation of up to 10–15 participants is estimated to enable saturation to be reached, as each participant will be an expert in the field of study. This estimation is also based on the researchers’ previous experience with qualitative research undertaken in the Emergency Department setting (Thomas et al. 2019).

5 Qualitative methods

The research methods will adopt qualitative inquiry to explore and gain an in-depth understanding of existing pathology ordering processes (Fig. 1 stage 1) as they occur in the ED environment prior to any development, perceived barriers and facilitators for the proposed tool (stage 1), and to gather comprehensive feedback on the ED POST prototype (Fig. 1 stage 2). Based on the researcher’s previous experience, semi-structured interviews and focus group qualitative methods can be effectively used in the Emergency Department setting to answer research questions (Thomas et al. 2019; Li et al. 2021).

5.1 Interviews and ‘think aloud’ observations

Data collection during stage 1 will be undertaken using semi-structured interviews and ‘think aloud’ observations with clinicians at the study site. Semi-structured interviews, based on open-ended interview questions (Given 2008a), will explore how pathology tests are ordered including the decision-making resources/information used to determine which tests are ordered. Interviews will also allow clinicians to individually provide feedback on perceived challenges and/or barriers to the proposed ED POST concept. The semi-structured interview guide will include the following questions:

  1. (I)

    Describe how a pathology test is ordered in the Emergency Department.

  2. (II)

    Describe what information/resources are used in the decision-making process for determining what tests should be ordered for a patient before or during the ordering process.

  3. (III)

    After a clinician enters patient variables at triage, the proposed ED-POST tool would return probability scores to assist with decision making on the pathology order for almost every possible presenting scenario. Would you find this type of information helpful?

  4. (IV)

    What challenges/barriers would you see in developing or implementing such a tool?

In addition, participants will be asked basic, non-identifying questions (e.g. age range, gender), and also asked to describe their current clinical role and years of experience within the Emergency Department.

Subsequent to interviews, ‘think aloud’ observations (Charters 2003) will be undertaken whereby clinicians are observed by a researcher whilst they perform pathology test ordering. Participants will be asked to explain their work and decision-making processes by “thinking aloud” as they place a pathology test order in the ED setting. Think aloud observations will be used to complement and triangulate interview data to explicate and demonstrate the processes and decision making associated with ordering pathology tests. Additional qualitative data will also be collected in the form of observational field notes and non-identifying photographs of relevant artefacts.

5.2 Demonstration and focus group

During the stage 2 prototype evaluation, participants will have the opportunity to view a demonstration of ED POST during a focus group session and provide feedback on the initial prototype. Qualitative data will be collected from the focus group discussions, which will be facilitated by a researcher, using semi-structured questions. Participants will be asked to provide feedback and input regarding the usability, utility, functionality and impact of the tool, as well as provide suggestions for improvement.

5.3 Data saturation

The concept of saturation in qualitative research has been described in general terms by Saunders et al. as “…a criterion for discontinuing data collection and/or analysis” (Saunders et al. 2018 p. 1894). Data saturation during the first stage of ED POST data collection will be determined by the researchers when pathology business processes (explored during interviews and observations) have been described in sufficient detail to inform the development of process models (including barriers/facilitators in the laboratory test ordering process). Data saturation during the second stage of ED POST data collection will be the point at which participants have no additional feedback to provide on the ED POST demonstration and no further improvement suggestions are offered.

6 Data analysis

Semi-structured interviews, think aloud observations and the focus group will be audio recorded for data capture. Audio recordings will be transcribed for inductive analysis. Transcripts will be analysed iteratively in NVivo qualitative data analysis software (QSR International, Melbourne, Australia, Version 20, 2022) and the principles of content analysis (Elo and Kyngäs 2008) and thematic analysis (Braun and Clarke 2006) will be applied to both identify themes and extract information to inform the development of business process workflow models.

6.1 Process modelling

Process models will be developed using structured analysis techniques including flow charts, workflow diagrams and/or using the Business Process Modelling and Notation 2.0.2 standard (Object Management Group 2014), which the researchers have previously used to develop process diagrams from qualitative data (Thomas et al. 2019). The information extracted from interview transcripts will be used to identify the tasks, decision points, roles and information requirements for the development of process models. The scope of modelling will commence with patient arrival at the Emergency Department and include all pathology ordering processes performed by Emergency Department clinicians. Process maps generated from stage 1 data will subsequently be reviewed in stage 2 to examine the impact of the prototype on pathology ordering workflows.

6.2 Quality and rigour

Establishing quality and rigor are important aspects of qualitative inquiry and approaches to validity and reliability will vary according to the context of the qualitative methodology used (Morse 2015; Hayashi Jr et al. 2021). Approaches to establishing rigour (including validity and reliability) have been included within the data collection and data analysis methods (Morse 2015; Coleman 2022). Triangulation (Given 2008b) has been included as an approach to establishing validity (Morse 2015) in stage 1 of the research and will involve data being collected through both semi-structured interviews and think-aloud observations. To achieve a ‘thick and rich’ data set (Morse 2015), interviews will proceed until data saturation is reached, audio recording will be employed to capture the entire interview discourse and observations will be used to complement interview findings. Furthermore, investigator triangulation will be used during stages 1 and 2 through the inclusion of two researchers in analysing the data. Member checking, described as “the method of returning an interview or analysed data to a participant…” (Birt et al. 2016 p. 1802), will be utilised for validation of work process flows (developed from raw data extracted across the entire dataset (Birt et al. 2016)) through draft diagrams being provided to participant/s at the study site to confirm their accuracy prior to prototype design and development.

The outcomes of the analysis of the qualitative data from stage 1 will inform the initial prototype design and development. Analyses of the qualitative data from the stage 2 evaluation will be used to inform revised development of the initial prototype. The outcomes of the study (whether positive or negative) will be reported in the literature.

7 Project governance

The ED POST project will be overseen by a two-tier governance structure involving a Project Steering Group (PSG- level 1) and a Project Management Team (PMT- level 2). The PSG will provide strategic level overview of the project activity, time, relationships, finance and outcomes. The PMT will undertake the day-to-day project activities to achieve project workplan outcomes on time. In addition to the governance committees, a stakeholder development group will also be formed (including pathology, research, technology/developer, and ED clinicians from up to 3 different sites) to ensure the incorporation of user needs through providing feedback, advice and clinical perspective on project activities during the development process.

8 Discussion

Qualitative methods have an important role in providing context-rich information to provide insight and meaning in health services research (Sofaer 1999). The methodological approach to the design and development of the ED POST prototype, as presented in detail in the current protocol, will draw on qualitative inquiry to gain a deep understanding of the pathology ordering process in the ED setting, and inform the development of business process models. Combining semi-structured interviews/think aloud observations and focus group methods to inform prototype design and development will provide both individual and collective clinician perspectives respectively. The pre-development and prototype evaluation stages of ED POST will emphasise close collaboration with ED clinicians with the aim of enhancing usability, utility and functionality of the proposed intervention through prototyping.

Human/user-centred approaches to CDSS prototype design have been reported in the literature (Beltrão et al. 2022; Hernandez et al. 2023), including Emergency Department settings (Ray et al. 2019; Jacobsohn et al. 2022; Vogel et al. 2022), studies using BPMN modelling (Vogel et al. 2022), and studies using interviews and/or observations (Ray et al. 2019; Beltrão et al. 2022). Such studies provide evidence of the positive value of incorporating human/user-centred design approaches in the prototype design and development process (Ray et al. 2019; Beltrão et al. 2022; Jacobsohn et al. 2022). Ray et al (2019) also identified important considerations during requirements gathering that may also be applicable to the Emergency Department setting in the current protocol, including different user needs across user groups and user concerns regarding workflow impacts (Ray et al. 2019). Cognisant of these findings, the approach detailed herein has included all clinicians (medical and nursing) who have responsibility for pathology test ordering in the ED, and includes a workflow focused design approach which will use process modelling to understand workflows prior to prototype development, and subsequently assess the impact of the design on existing work practices.

The protocol for the current study is based on an Emergency Department setting in the Australian healthcare context and this constraint may impact the generalisability of the study findings to other hospitals or healthcare settings. In an attempt to reduce the impact of this limitation, ED clinicians from additional public hospitals will be invited to attend the prototype demonstration. The inclusion of these external stakeholders will ensure representation beyond the study site when feedback is sought on the prototype design. Measures to reduce the limitations of qualitative research and to enhance quality and rigor have been incorporated into the study protocol design, including data and investigator triangulation, and member checking of draft workflows. During the conduct of the study, any specific limitations that are encountered during the research will be recorded and discussed in the reporting of results. The multi-method qualitative approach serves to inform the design and development of a dynamic and fit-for-purpose clinical decision support tool to address unwanted variation in laboratory test ordering, and the reporting of outcomes from the study will include any challenges or shortcomings experienced from the application of the protocol. Health researchers and digital health developers can draw on the approach detailed herein to inform research planning activities for health information technology intervention design projects. The outcomes of the development of ED POST, as presented in this research protocol, will be reported in the literature.