Introduction

March 16th, 2017, New York, NY:Yadira Arroyo, a 14-year veteran of FDNY was killed by a career criminal who commandeered her ambulance, ran her over, and dragged her beneath its wheels. Encountering violence in the field is not a rare occurrence for EMTs. In New York City, more than 100 members a year are assaulted on the job according to Robert Ungar, a spokesman for the Uniformed EMTs, Paramedics & Fire Inspectors F.D.N.Y. union (Mueller 2017).

May 1st, 2017, Dallas, TX:Paramedics responded to a call that purported a suicidal patient. Upon arriving on scene, EMS responders discovered a gunshot victim laying in the street and began to administer care. While providing life saving measures to the gunshot victim, the paramedic providing treatment was critically wounded by the armed gunman. The injured medic was in critical but stable condition after undergoing lifesaving surgery. The medic will require multiple surgeries and extensive treatment before a full recovery (McLaughlin 2017).

There were 29 million calls for Emergency Medical Services (EMS) in 2015, a 23% increase from 2014 (National EMS Information System 2016). This increase represents a continually growing trend in the United States. Fire departments provide about 40% of the nation’s EMS services (National Association of State EMS Officials 2011). In 2015, on average 64% of fire department 911 calls were for medical emergencies (National Fire Protection Association 2016), but can run as high as 90% (FIREHOUSE 2016).

Paramedics and EMTs believe they are under severe stress and are concerned about impacts on their mental health (Everline 2016; J. A. Taylor et al. 2016). However, such perceptions have not yet been systematically measured. In a previous study on their experiences with patient-initiated violence the following recollections were reported (J. A. Taylor et al. 2016):

“I have been kicked, punched, bitten, spit on, verbally abused. You name it, I’ve had it all.”

“And [I] went to court, and this is where it's disheartening, because that's supposed to be felony assault…. And I'm wasting my time going to court two and three times...I knew there was no confidence in the system...I mean, you shouldn't be able to do that to someone who's trying to help you. Felony assault should stick.”

The annual rate of non-fatal injuries among U.S. paramedics is five times higher than the national average for all workers (B. J. Maguire et al. 2005). The annual rate of occupational fatalities among paramedics is two times higher than the national average for all workers (Brian J. Maguire et al. 2002). In a retrospective cohort study of nationally registered U.S. EMTs, 8 % of fatalities were due to assaults (B. J. Maguire and Smith 2013). Compared to health care settings such as hospitals, workplace violence in EMS is inadequately described and requires further consideration (Koritsas et al. 2009; Koritsas et al. 2007; Terry Kowalenko et al. 2013; T. Kowalenko et al. 2005).

The National Institute for Occupational Safety and Health (NIOSH) lists 10 factors that may increase a worker’s risk for workplace assault, such as contact with the public, having a mobile workplace, working with unstable or volatile persons, working alone or in small numbers, etc. Eight factors apply to the EMS work environment (Jacob 2004). In an analysis of near-miss and injury events reported during EMS runs to the National Fire Fighter Near-Miss Reporting System, the most commonly identified mechanism of injury was assaults. EMS responders were threatened or assaulted by patients, family members, and bystanders. Patient factors included: drug/alcohol intoxication, mental illness, and underlying health conditions (e.g. seizure, hypoglycemia)(J. A. Taylor et al. 2015). In a study on fire service injury, paramedics experienced a 14-fold higher odds for violence injuries than firefighters (J. A. Taylor et al. 2016).

In a study of safety climate in fire departments across the United States, firefighters reported that the 911 system is strained due to the high volume of low acuity medical calls that occupy much of their workload, which divert resources from true emergencies and lead to unwarranted occupational hazards like speeding to respond to non-serious calls. As a result, firefighters reported high occupational stress, low morale, and a desensitization to community needs. Firefighters called for improvements to the 911 system including better triage, more targeted use of EMS resources, continuing education to align with job demands, and a strengthened social safety net to address the persistent needs of poor and elderly populations (Cannuscio et al. 2016). A critical barrier to progress in fire-based EMS is understanding the organizational, mental health, and safety burden that providers currently carry as they respond to an increasing community demand for services. There are concerns that this workforce is experiencing poor safety and organizational outcomes such as injury, depression, anxiety, PTSD, burnout, and decreasing job satisfaction. EMS responders have also reported higher rates of suicidal ideation and suicide attempts (Stanley et al. 2015; Stanley et al. 2016). These outcomes are very costly to employers and threaten the stability of the healthcare safety net EMS provides (Bigham et al. 2014; Federiuk 1992; Nordberg 1992; Patterson et al. 2010).

A systematic literature review of violence against firefighters and EMS responders in peer-reviewed and industrial journals (1978–2016) found that between 57 and 93% of EMS responders reported having experienced an act of verbal and/or physical violence at least once in their career (Taylor and Murray 2017). Up to the early 1990s, the issue of violence towards EMS responders was only discussed within industrial journals, underscoring the fact that industry understood the risks of the job long before the first peer-reviewed research was published in 1993. While prevalence estimates fluctuate slightly, authors are discussing the same issues 40 years later. No evidence-based interventions have been developed to support EMS responders from violence on the job, however industry articles provide potential organizational improvements that could mitigate the risks of assault including: policies for police backup; better information from the dispatch system including flags to identify previously violent locations; improved cultural competency, de-escalation, and body language techniques; personal protective equipment; and improved reporting of assaults (Nethercott 1997).

Given these best practices, the utility of a checklist that organizations can use to support EMS responders is plausible. Checklists acknowledge that accidents are inevitable in complex systems and work to reduce complexity (Perrow 1984). They have been used successfully in a myriad of industries to reduce errors, inconsistencies, and unsafe practice. For example, the field of aviation made strides in safety through the use of pilot checklists. From 1980 to 1996, 25% of fatal approach and landing crashes were due to performing the wrong action or omitting critical actions (Ashford 1998). To reduce such errors, pilots have access to brief but easy to read checklists such as the Northwest Airlines MD - 80 checklist (National Aeronautics and Space Administration 1990). Thus, the percentage of accidents due to human error decreased from 90 to 55 (Shappell et al. 2006). Other well-regarded checklists include: the Toyota Production System - a systems approach to define processes and needed communication coupled together to reduce confusion and encourage a culture of learning (de Bucourt et al. 2011; Institute of Medicine 2012; Spear and Bowen 1999; Toyota 2015) and the “Doctor’s Checklist” created at Johns Hopkins Hospital to avoid infections when inserting central lines into ICU patients (Berenholtz et al. 2004).

In EMS, there are numerous checklists for each phase of the call, such as the Responding to Violence Checklist by EMS Health & Safety (Collopy et al. 2011) or the Aggression Continuum by Steven Wilder and Chris Sorensen (Vernon 2009). None of the common practices were being codified into any standardized training at fire departments, meaning that first responders would need to self-educate in order to equip themselves with these skills. This gap in training and policy, coupled with the concerning levels of stress and violence experienced in EMS, initiated the need for development of a systems-level checklist.

The activity described herein is part of a Federal Emergency Management Agency (FEMA)-funded study, “Stress and Violence to Fire-based EMS Responders (SAVER).” The first of four study aims was to develop a systems-level checklist for violence against fire-based EMS responders using: (1) findings from a systematic review of academic and industrial literature and (2) outcomes from a national stakeholder meeting during which consensus on checklist items was achieved.

Creation of the SAVER Checklist

The World Health Organization’s Patient Safety Programme developed the Surgical Safety Checklist using steps adapted from aviation (Weiser et al. 2010). The SAVER checklist, tailored to fire-based EMS responders, followed a similar development process including the steps of content and format, timing, trial and feedback (Weiser et al. 2010). Formal testing and evaluation, and local modification will be evaluated in future activities with fire departments participating in the SAVER program (Weiser et al. 2010).

The Checklist Manifesto by Atul Gawande (Gawande 2010) was a conceptual resource for the development of the SAVER systems-level checklist. However, most checklists currently in use – whether in healthcare (Weiser et al. 2010), nuclear power operations (Jou et al. 2009), aviation (Federal Avitation Administration 1981), or other high-risk industries – are focused on the individual, rather than on the system in which they operate:

Rather than being the main instigators of an accident, operators tend to be the inheritors of system defects. . . . Their part is that of adding the final garnish to a lethal brew that has been long in the cooking (Reason 1990).

In the absence of systems thinking, there has been a tradition of blaming workers for their injuries. Even in cases of assault by patients, the fire and rescue service has few answers other than punishing the patient (e.g., felony conviction) or putting the onus back on the responder (“be careful,” “practice scene safety,” “it’s part of the job”). Systems thinking instructs designers to assume adverse events will happen and anticipate opportunities for prevention as upstream as possible from the interactions that cause harm. Such upstream opportunities are largely focused on organizational policies, procedures, and practices – as opposed to individual actions (Reason 2000). Based on our continuing work with the fire and rescue service, we wanted to propose a solution to work-related assaults that removes the burden from the individual-level responder and places it back on the level responsible for safety and health – the department and labor union. It is these organizations that make policy and create opportunities for training. Individual firefighters and medics do not. In our study of checklists, it seemed possible that a checklist could have benefit at the organizational level.

This is not to say that individuals do not have a role in preventing adverse events such as workplace violence. EMS responders are notoriously overworked with little time to think about organizational interventions (Cannuscio et al. 2016; J. A. Taylor et al. 2016). In addition, the hierarchical work environment of the fire and rescue service can make it hard for individuals to raise their voices. Gawande’s guidance to “push the power of decision making out to the periphery and away from the center” by employing checklists, ensures that the individuals with the least amount of power within an organization have the authority to make autonomous decisions that support their mission and safety (Gawande 2010). This translates into pause points - a set of discreet criteria that must be met before proceeding to the next task. A common example is the use of a “time out” at the beginning of a surgery whereby anyone on the team may raise concerns for the patient and the procedure about to be performed, which are then addressed before continuing (Gawande 2010).

Methods

Development of Candidate Checklist Items

The primary literature review of violence against EMS responders was conducted previously for a commissioned report (Taylor and Murray 2017). That review had three phases. Phase 1 included articles reviewed based on title, abstract, and keywords; Phase 2 included article review, assessment, and documentation based on titles and abstracts; and Phase 3 included an in-depth analysis of all industrial and academic literature on the topic. Of 386 articles described in that report, a subset categorized as “preparedness and intervention” from industrial sources only (n = 144) were selected because they described best practices, policies, and training opportunities to prevent violence against EMS responders (Fig. 1). It was from these sources that the idea of a systems-level checklist emerged.

Fig. 1
figure 1

Flow Chart of Total Industrial and Academic Articles used for SAVER Systems Checklist

Full size image

A subsequent search of the academic literature was conducted to describe the development, application, and evaluation of checklists. Search terms included: surgical checklist, aviation checklist, Atul Gawande, safety culture checklist, systems checklist, and safety domains. Of the 25 academic articles collected during this secondary literature search, 18 articles were selected for inclusion by CKH and RMM, because they discussed the development of systems checklists and organizational-level applications (Fig. 1).

From the above described literature reviews, a total of 162 manuscripts were compiled, read, and then analyzed chronologically during a five-day research retreat (CKH, LJS, ALD, RMM, and JAT). Further inclusion and exclusion criteria were discussed as a group to establish inter-rater reliability. Literature that did not suggest ideas for prevention that could be developed into a candidate checklist item were excluded after consultation with JAT and confirmation by RMM (n = 40). 122 articles were referenced for the development of candidate SAVER systems checklist items (Fig. 1).

Upon the conclusion of the research retreat, 318 candidate checklist items were developed. Each candidate checklist item was documented and assigned to the relevant phase of EMS response: “traveling to the event”, “scene arrival/prior to entry”, “patient care”, transport to the hospital”, “transfer to emergency department staff”, and “assessing readiness to return to service”. Next, Q-methodology, a systematic process for assessing subjective viewpoints often used in psychology and qualitative research, was employed. Q-methodology assists with the identification of similarities and patterns among sources, as well as their relationship to broader statements and categories (Shinebourne 2009). Each candidate checklist item was reviewed, the source verified, and then de-duplicated. Similar checklist items were grouped together based on content and co-author consensus. A total of 39 emergent themes were cataloged according to their goodness of fit within its assigned phase of EMS response or reassigned to the relevant phase. Then, all candidate checklist items were re-evaluated for goodness of fit within each phase, and each theme.

Next, checklist items were focused at the level through which an organization (i.e., fire department and labor union) could support EMS responders. Items were drafted into one of four levels that emanated from the EMS literature: policy, training, technological or engineering modifications, and individual-level actions. From the initial 318 candidate items, a total of 159 checklist items and six pause points were developed. Themes were further refined based upon the development of the draft checklist items and team consensus (reduced from n = 39 to n = 25). Of the 159 checklist items, there were 78 policy items, 59 training items, 6 technological or engineering items, 16 individual-level items across the six phases of the checklist.

Stakeholder Consensus Conference/Resultant Checklist

To provide the opportunity to review and revise the candidate checklist items, 41 diverse subject matter experts (SMEs) representing 27 different affiliations were invited to a two-day SAVER “Systems-level Checklist Consensus Conference (SC3)” in July 2018. The 41 SMEs included: 12 national fire service and EMS organizations, 6 individuals from federal, state, and local governments, 3 research and academic organizations, 14 fire department members, and 6 labor union representatives. In the United States, fire department labor unions are dues-paying membership organizations that advocate for firefighter and EMS responder employment benefits (i.e., work hours, wages, health and safety conditions) through collective bargaining processes. To ensure organizational and occupational representation from each of the four participating fire departments serving as SAVER study sites, one individual from each of the following fire department levels was invited: leadership, labor union, EMS field supervisor, and paramedic with 10–15 years of experience. Representatives from dispatch and law enforcement were also invited, though only 1 dispatch representative was able to attend. None of the attendees had previous knowledge of the items in any phase of the checklist, though all received a one-page document describing the SAVER program. SMEs were 78% male and 22% female.

We deployed a facilitated consensus-building collaboration method using a series of focus groups structured around three separate ThinkLet systems (De Vreede et al. 2006). The ThinkLet guides are available in the appendix and details on the ThinkLet process deployed are available in Fig. 2. As ThinkLet sessions were recorded, a human subjects protocol was submitted and approved by the Drexel University Institutional Review Board.

Fig. 2
figure 2

ThinkLet Process

Full size image

Through the ThinkLet process, the checklist was reviewed, revised, updated, and finally rated for feasibility. Following ThinkLets 1–3, the checklist grew from 159 to 242 candidate items. These items were reviewed by the co-authors who served as the facilitation team and preparations made to include all newly generated checklist items in ThinkLet 4 at the start of the next day. ThinkLet 4 processes led to the deletion of 10 candidate checklist items based on SME consensus. The resultant 232 candidate checklist items were voted on based on feasibility using three criteria: most feasible (MF), less feasible (LF), and extremely difficult (ED) (Figure 3). The feasibility assessment was completed individually to remove the potential for social desirability. Percent MF, percent LF, percent ED, and percent missing votes were calculated by total possible respondents and by each phase. For example, percent missing was calculated by the number of missing votes per checklist item, divided by the total possible responses (41) and divided by the total number of checklist items per that phase, multiplied by 100.

Fig. 3
figure 3

Rating Scale for Candidate Checklist Items

Full size image

Further refinements to the resultant checklist were completed by JAT, RMM, and JAA. The resultant checklist was reviewed for grammar, inconsistencies, and redundancies and revised accordingly.

Results

Development of Candidate Checklist Items

The literature review process yielded 159 checklist items organized by phase of EMS response: “traveling to the scene,” “scene arrival,” “patient care,” and “assessing readiness to return to service.” Due to the paucity of literature and checklist items in the “transport to the hospital” and “transfer to emergency department staff” phases, these items were subsumed into the “patient care” phase. Two additional phases, “pre-event” and “post-event” were developed based on the team’s analysis of gaps in the literature and results from the co-authors’ previous EMS research.

The “pre-event” phase focuses on the organizational structure (e.g., a fire department and its labor union). It is intended to help these organizations work together to develop policy and create training curricula that address EMS stress and violence. This enables and supports two goals: safe workers and quality patient care. For example (Table 2):

  • 1.3: Does your department train EMS responders for potential verbal and physical violence (examples include: prevention, patient abandonment, felonious assault laws, cultural competency, simulation, self-defense, law enforcement cross-training, fit-for-duty, etc.)?

  • 1.5: Does your department express through policy that verbal and physical violence against members is not tolerated?

The “post-event” phase emerged from the realization that while there are many tasks that need to be completed after an EMS run (e.g., typing patient care reports, cleaning and restocking the ambulance, communicating availability to dispatch, bathroom, and food breaks), there was no point in the phases of response that gave EMS responders protected time to document and report violent encounters, or reach out for mental health support. If encounters are not documented, evidence to describe the extent of the stress and violence problem does not exist. Furthermore, opportunities for recovery and treatment of physical and psychological injuries cannot be addressed. For this reason, the post-event phase has a strong emphasis on mental health impact and recovery, as well as a feedback mechanism for what is not working in the field. For example (Table 2):

  • 6.1.f: Does your department have a policy that protects an EMS responder’s time - either by going out of service or using overtime - so that they can easily report any acts of violence or exposure that they experienced on a call, before they return to service and go on their next call?

  • 6.15: Does your department’s training curriculum recognize and train on stress as a chronic occupational exposure including the relationship between the EMS responder workload and its cumulative stress impact?

Following the consensus conference, these new phases of emergency response remained on the checklist, confirming need, acceptance, and relevance to EMS work by the SMEs.

Pause points

Using the previously described systems framework, draft checklist items were developed at the following levels within each phase of response: policy, training, technological or engineering modifications, and individual-level actions. Few individual-level actions were identified. Six of these were designated as pause points (Gawande 2010) because they focused on potential individual actions that could be taken at critical decision points to protect the responder’s safety. Pause points provide responders with the authority and autonomy needed to engage in safe patient care, provide a protection of self, and engineer an opportunity to give feedback to the organization about what is not working in the field. This results in a flipping of the traditional paradigm from placing the onus for safety on the responder and moving it back onto the organization:

  • Traveling to the scene: If you have knowledge that this is a previously known violent location, request and wait for law enforcement backup.

  • Scene arrival: Before exiting the ambulance, are all of the resources you need in place to safely begin patient care?

  • Patient care: Before transport, does your patient require restraint and have they been checked for weapons?

  • Assessing readiness to return to service: Are you mentally and physically ready to return to service?

  • Post-event: If you have experienced verbal or physical violence, have you utilized the appropriate method for reporting?

  • Post-event: Have you sought and received the physical and long-term mental health resources you feel will enable you to return to work whole and ready?

Outcomes from the SAVER Systems-Level Checklist Consensus Conference (SC3)

Feasibility Assessment

Overall, 47.5% of votes across all phases rated items as most feasible, 33.7% as less feasible, and 11.6% as extremely difficult. (Fig. 4, Table 1). “Assessing readiness to return to service” was the only phase in which the majority of votes rated items as less feasible (39.8%). This phase also had the highest percentage of votes rating items as extremely difficult (19.8%). Missing votes across the phases ranged from 5.3–13.3%.

Fig. 4
figure 4

Checklist Feasibility Distribution by Number of Votes

Full size image
Table 1 Feasibility assessment of checklist items (n = votes)
Full size table

The final SAVER checklist (Table 2) contains 174 items organized by six phases of EMS response: “pre-event” (n = 41), “traveling to the scene” (n = 16), “scene arrival” (n = 14), “patient care” (n = 30), “assessing readiness to return to service” (n = 30), and “post-event” (n = 43). Pause points (n = 6) for the individual EMS responder were incorporated at the end of each of phase. Analysis of the final checklist shows that 70% of the items were from the original candidate items developed by the research team during the literature review process, and 30% of the items came from new items generated at the conference. Thus, the final checklist as reported here includes comprehensive scientific and industrial literature as well as integrated subject-matter expertise from the field.

Table 2 “SAVER Systems-level Checklist”
Full size table

Discussion

High-reliability organizations, such as EMS, are expected to operate in complex environments that have a high risk for occupational illness and injury (Weick and Sutcliffe 2007). The risks that an EMS provider is exposed to on each run has the potential to be career or life-ending. The increasing number of calls in the United States adds to an already significant emotional demand inherent to health care professions. At the systems-level, organizations have a responsibility to develop policies and training that support and confer competencies necessary to effectively perform the job of an EMS responder. The SAVER systems-level Checklist addresses such opportunities by pointing organizations to the resources, tools, and knowledge they need to support the ever-changing mobile work environment faced by their EMS responders. The SAVER Checklist has two goals: (1) fire departments and labor unions should immediately use the systems-level checklist to assess and then plan for implementation of the items – starting with what is most feasible, and (2) these organizations should empower EMS responders to use the pause points (individual-level checklist) to protect their health and safety while on calls. The organizations should then act on feedback emanating from members in the field. To uphold the importance of EMS responder feedback, organizational leaders are to ask the EMS responders what the facilitators and barriers to effective implementation are for each checklist item.

The SAVER systems-level Checklist took what industry journals have been saying for 40 years - that violent events to first responders can be mitigated - and created a checklist for the system, instead of a checklist which would put more burden on already overstretched responders.

The SAVER checklist is innovative and different from other checklists because of its focus at the organizational - rather than the individual - level. If organizations are truly concerned about the health and safety of their members, they will demonstrate such through the development of meaningful policy and ongoing training. The individual-level pause points confer autonomy and authority to individual responders that allow them to make decisions that prioritize their safety. In so doing, the organization is creating a supportive environment that can further positively impact organizational outcomes such as burnout, job engagement, and job satisfaction, as well as decreasing the number of assaults and injuries experienced by EMS personnel.

The feasibility results from the consensus conference provided interesting findings, some of which surprised our research team. For example, 47% of the votes rated items as most feasible signifying that our diverse SMEs believe that these checklist items could be implemented in 3–6 months. That 33% of the remaining votes rated items as less feasible, but feasible within 1–2 years was also encouraging. These assessments can help fire departments prioritize the most feasible items and then move on as time and resources allow to those rated as less-feasible.

The “pre-event” through “patient care” phases were rated as most feasible. Pre-event is a new phase of the response process developed by our research team and underscores the importance of the structure of an organization, specifically, a fire department. What policies, procedures, practices, and training opportunities should be present in order to ensure the health and safety of EMS responders? What technology or engineering innovations are needed in order to keep them safe on the job? And what can the organization do to ensure their safety and health while they are on a call so that they can focus on quality patient care?

All pre-event items originally generated by the authors during the literature review phase were included in the final checklist (n = 22), along with an almost equal number of new items developed by the conference participants (n = 19). Fifty one percent of the votes described items in this phase as most feasible, which is inspiring since they are largely focused on policy development. We would have expected less enthusiasm more in line with that expressed in the “assessing readiness” and “post-event” categories. If fire departments and their labor unions can focus their efforts in the pre-event phase where they rate the action items to be most feasible, then there is a potential for mitigating even initial exposure to violence for EMS responders.

Development of the pre-event phase affirmed our use of a systems approach in that there are policies, procedures, and practices that departments can create before an emergency response even begins. Pre-event is supported by research on safety climate which stresses upstream intervention in the form of policy and training that will reward and support individual safety behaviors and therefore prevent undesirable outcomes like occupational injury (Beus and Payne 2010; Christian et al. 2009; Huang et al. 2016; Nahrgang et al. 2011; Zohar 2010). The more organizations focus on the pre-event phase, the more focused they are on prevention.

The last phase of the checklist, the post-event, was developed by our team as well. Traditionally departmental policies and procedures have emphasized what responders should do when on a call itself, but after the job is done there are many tasks that should be completed to return EMS responders back to the field whole and ready for the next call. For example, reporting injuries, assaults, or experiences of verbal violence to supervisors; the need for mental health support and rest and recovery from this highly emotionally demanding job; and a feedback mechanism for what is not working in the field. As the post-event phase is focused on protected time for reporting and pursuit of mental health resources, additional momentum and cultural shifts will need to continue in order for the fire service to fully embrace this phase of the checklist (see NLSI #12 and 13 (National Fallen Firefighters Foundation 2004)).

The phases “assessing readiness to return to service” and the “post-event” phases were rated as less feasible. Given that these phases had little to no evidence in the academic or industrial literature, it is not surprising that our SMEs rated these items as “extremely difficult” more so than other categories. These phases likely represent where the current system is breaking down or needs more development. This is simply part of continuing the fire service evolution from that of a reactionary culture (i.e., waiting until something bad happens to address safety concerns) toward an increasingly proactive culture. These items are largely more difficult to implement because they involve organizational changes, behavioral changes, increases in staffing, and additional resources that are already stretched thin in most fire and rescue departments. To this point, because the checklist’s length could present a realistic impediment, we suggest that fire departments start with what is most feasible and then plan for less feasible and extremely difficult items to be addressed over time. We are also investigating reducing the amount of items in each phase of the checklist by selecting the items that most support operationalization of the pause points as initiators for organizational change.

Limitations

Although the development of the checklist was an essential first step to improving the safety and well-being of EMS responders, the development process was not without limitations. First, one goal of the consensus conference using the ThinkLet process was to reach data saturation on the potential checklist items. Further, our team selected three rounds of Item Generation ThinkLets due to time constraints within a two-day conference format and supported by data saturation science, specific to qualitative research methods and ThinkLets (Steinhauser et al. 2008; Tracy 2013). Though the number of new ideas toward the end were fewer and we feel we were close to saturation, additional ThinkLet sessions would have reached complete saturation. We are confident that what is presented is comprehensive and complete.

Second, the purpose of ThinkLets 1–3 was to generate ideas not present or missing from the draft checklist. As such, these newly generated ideas were generally not written into formal checklist items. While time between Day 1 and 2 allowed the authors to review and incorporate the newly generated checklist ideas, time constraints limited the authors’ ability to significantly process and refine all checklist ideas into formal, polished checklist items (a process candidate items enjoyed). Thus, as a check to verify that the new items were not rated as either more (e.g. because the SMEs like them) or less (e.g. because they were not as polished) feasible was performed. The overall feasibility results for the original items versus the new items from the conference did not differ significantly. That is, the pattern of most feasible, less feasible, and extremely difficult was essentially the same for both groups of items on the final checklist (i.e. 50%, 30%, 12%, with 8% missing; data not shown, but on request available from the first author). This helped us feel confident that biases in the ratings were neither present in the feasibility assessment, nor in the final checklist as currently constituted.

Third, due to unexpected resource limitations related to ThinkLet 5, we adjusted the process and implementation of the feasibility assessment. Specifically, instead of open voting visible to everyone at the conference, we initiated a closed ballot approach. Interestingly, this conferred a strength, in that it eliminated the potential for participant responses to be altered by social desirability when discussed in a group setting; however, because these new processes were not planned nor rehearsed, opportunities to field test and revise the formatting of the checklist were missed, and may have contributed to the resulting missing data from ThinkLet 5 (i.e. participants could skip rating items where desired). Other potential reasons for missing data include: time constraints and phrasing of some checklist items as an “if/then” statement; if the stem item was not voted with a high regard to feasibility, then the subsequent “if/then” root item was largely skipped.

Fourth, while the modified feasibility assessment in ThinkLet 5 was re-designed to be anonymous, in retrospect we believe it would have been beneficial to capture the participants’ identity during this ThinkLet session. Without these data, we are unable to stratify responses based on organization or rank within a fire department. It is possible that members of fire departments were more likely to rate feasibility with more certainty than others from outside the fire service. Thus, our ability to make conclusions from the feasibility assessment that would be impactful for checklist implementation processes are limited. To address this issue, we will conduct an additional feasibility assessment prior to checklist implementation at our study sites. Moreover, because the primary focus of the ThinkLet process was to establish consensus on the feasibility of the checklist, we did not assess participants’ valuation regarding the criticality or importance of each checklist item. Future work will include asking our study sites to review and rate the checklist items by the importance of the item for implementation and see how these ratings relate to feasibility. We will also measure if any checklist items are currently in use. Despite these limitations, we are confident our actualized methods of ThinkLet 5 resulted in an improved and streamlined process.

Finally, our exceptional panel of SMEs did not include some of the expert perspectives we had hoped to include, such as more 911 dispatchers and law enforcement. Despite this, we feel confident that key stakeholders perspectives were represented in the process of developing the checklist shown here. Through this publication, we invite those who have ideas and feedback on the checklist to contact the corresponding author.

Future Use, Implementation and Validation

By utilizing the highly efficient ThinkLet process, key SMEs in the field of EMS were able to come to consensus within a two-day conference format to ensure the SAVER systems-level checklist was comprehensive for all phases of EMS response. With a focus on systems-level actions, the checklist will require leaders and representatives in both fire departments and labor unions to work collaboratively to maximize the acceptability and impact of the checklist upon implementation. The checklist should be used to promote dialogue and goal setting over time. As all checklist items have been developed within a systems framework, organizations must assess the facilitators and barriers to implementing each checklist item and incorporate this into their plan. This important dialogue will be crucial to the overall impact and support felt by the front-line providers.

The SAVER systems-level Checklist is an innovative application of traditional checklists, designed to shift the onus of safety and health from that of the individual first responder to the organization. The individual EMS responder does not use the systems level checklist. The only thing they need to do is use the pause points (an individual-level checklist) to provide feedback and stop ongoing processes from hurting them. Therefore, while the organization is responsible for 174 checklist items on the resultant SAVER checklist, the individual EMS responder is responsible only for 6. The SAVER checklist is predominately focused on actions that the leadership team (the fire department and labor union) can institute through training, policy, and environmental modifications. The pause points act as feedback loops for the individual EMS responder. This distributes the power throughout the hierarchy of the organization and gives the individual responder the autonomy and authority to pause mid-response at multiple time points, should they determine there are risks or threats to their personal safety. The training and policy instituted at the department-level will give first responders the tools to know how and when to call a ‘time out’ at these specific pause points to maintain their safety while responding. It is interesting that when the 174 final checklist items were redistributed among the domains of policy, training, environment, and individual, all of the original numbers stayed the same with the exception of policy. That category increased from the 78 draft items to 109 after the subject matter experts gave their input. The authors believe this is an expression of the importance of policy as an emergent need in the United States fire service.

While the checklist was developed with fire-based EMS organizations in mind, it holds relevancy to the private EMS sector, as well as small, rural, and volunteer departments. Representatives from these sectors were present at the consensus conference, thus we are confident that the checklist will have impact for various models of EMS. Upon completion of an additional feasibility assessment with the fire department study sites, the checklist will be tested with four large-metropolitan fire-based EMS departments. This implementation will include a battery of psychological tests to examine the degree to which the organizational-level intervention is impacting the well-being of employees. The implications for EMS responders are clear in that if the intervention works as designed, we would expect to see decreasing levels of burnout, decreasing assaults and injuries, and increasing job satisfaction and engagement with work. So, while the checklist is organizational in implementation, the result is impact on the individual worker.

This checklist was the result of a significant collaboration between the subject matter experts and the research team. In addition to understanding what is immediately achievable versus what remains long-term, it is important to acknowledge the emotions driving this work. The EMS side of fire has long been ignored. Its hazards from violence poorly described and tracked. Therefore, the importance of this work to our SMEs is best memorialized in their own words: “It’s about time someone cared enough to do something!”, “I feel this is the turning point to provide funding, resources, and a voice to people who need it.”, and “This subject has been long overdue. The environment is not getting any safer, the streets are getting more dangerous.”