Keywords

1 Introduction

1.1 Risk-Informed Safety Margin Characterization (RISMC)

The Risk-Informed Safety Margin Characterization (RISMC) Pathway within the United States (U.S.) Department of Energy’s (DOE’s) Light Water Reactor Sustainability (LWRS) Program develops and delivers approaches to manage safety margins for nuclear power plants [1, 2]. This important information supports the nuclear power plant owner/operator decision-making associated with near and long-term operation. The RISMC approach can optimize plant safety and performance by incorporating a novel interaction between probabilistic risk simulation and mechanistic codes for plant-level physics. The new functionality allows the risk simulation module to serve as a “scenario generator” that feeds information to the mechanistic codes.

When evaluating the safety margin, what we want to understand is not just the frequency of an event like core damage, but how close we are (or are not) to key safety-related events and how might we increase our safety margin. The RISMC Pathway uses a probabilistic margin approach to quantify impacts to reliability and safety. As part of the quantification, we use both probabilistic (via risk simulation) and mechanistic (via physics models) approaches. Safety margin and uncertainty quantification rely on plant physics (e.g., thermal-hydraulics and reactor kinetics) coupled with probabilistic risk simulation. The coupling takes place through the interchange of physical parameters (e.g., node pressure) and operational or accident scenarios.

In order to perform advanced safety analysis, the RISMC project has a toolkit that was developed internally at Idaho National Laboratory (INL) using the Multiphysics Object Oriented Simulation Environment (MOOSE) [3] as the underlying numerical solver framework. This toolkit consists of the several software tools, which include:

  • Reactor Excursion and Leak Analysis Program (RELAP-7) [4]: the code responsible for simulating the thermal-hydraulic dynamics of the plant.

  • Reactor Analysis and Virtual Control Environment (RAVEN) [5]: it has two main functions: (1) act as a controller of the RELAP-7 simulation and (2) generate multiple scenarios (i.e., a sampler) by stochastically changing the order and/or timing of events.

1.2 Human Reliability Modeling within RISMC

In past RISMC studies, human interactions have been modeled in a simplified manner. We used the method described in [6] to model human related actions, which are based on the Standardized Plant Analysis Risk-Human Reliability Analysis (SPAR-H) model [7] contained in Systems Analysis Programs for Hands-on Integrated Reliability Evaluations (SAPHIRE). The SPAR-H model characterizes each operator action through eight parameters called performance shaping factors (PSFs) that are used to compute the probability that an action will happen or not; the probability values are then inserted into the fault and event trees that contain such events.

However, from a simulation point of view we are not seeking to determine if an action is performed but rather when such an action is performed. Thus, we need a probability distribution function (pdf) that defines the probability that a human related action occurs as a function of time. Because most human reliability analysis (HRA) methods like SPAR-H do not provide a dynamic account of human actions or the PSFs, it is desirable to review modeling outside HRA such as human performance modeling.

1.3 Importance of HRA Simulation Approaches

Cacciabue [8] and others (e.g., [9]) have outlined the importance of simulation and modeling of human performance for the field of HRA. Specifically, simulation and modeling address the dynamic nature of human performance in a way that has not been found in most HRA methods. Concurrent to the emergence of simulation and modeling, several authors (e.g., [10, 11]) have posited the need for dynamic HRA and have begun developing new HRA methods or modifying existing HRA methods to account for the dynamic progression of human behavior leading up to and following human failure events. Currently, there is strong interest in the fusion of simulation and modeling with HRA (e.g., [12–16]).

2 Traditional Static HRA Methods

Swain [17], Hollnagel [18], and Boring [19] state that the practice of HRA started in the 1950s, with the first symposiums meeting in the 1960s, and that the development of formal HRA engineering methods started in the ensuing years when it was applied to modeling human performance in the construction of nuclear weapons and nuclear power reactors. The seminal HRA method for nuclear energy, a Technique for Human Error Rate Prediction (THERP) [20] was developed during this time, and its final version was developed partly as a response to the accident at Three Mile Island. Since the 1980s, there has been a proliferation of HRA methods, which have attempted to improve various aspects of early methods. Which aspects of the early methods were addressed depended greatly on the developers of the new method and the issue or issues they were trying to address.

Boring et al. [21] summarized past attempts to compare and categorize HRA methods, and noted that many of those attempts ended up with complex and non-orthogonal schemes due to the number and variety of published HRA methods. One example of a fairly complex summary is Chandler et al. [22], which compared many HRA methods across multiple dimensions, including the methods’:

  • Features and capabilities;

  • Source (i.e., technical basis and/or data basis), approach, and treatment of dependencies and recovery;

  • Error identification and human error probability (HEP) estimation approach;

  • Resource requirements; and

  • Cost and availability of method, tools, and data.

NUREG-1842 [23], and more recently NUREG-2127 [24] also provide comprehensive comparative summaries of HRA methods used by the nuclear industry. The reader is encouraged to review these categorization schemes to understand the various approaches to grouping and comparing HRA methods.

2.1 Shortcomings of Non-simulation Based HRA

As Hollnagel [18] has pointed out, non-simulation based HRA methods are essentially modeled after traditional probabilistic reliability assessment (PRA). That is, these methods use the same basic approach PRA uses to model equipment reliability, with two key exceptions. The first is that the modeling of equipment failures is replaced by modeling human failures at tasks and/or activities. The second exception is that wider uncertainty bands are used to account for the increased variability in human performance relative to equipment performance, which is often attributed to individual differences between people, the time-dependent nature of many human tasks, and the non-orthogonality of factors that influence human performance. Inherent to this approach, given the assumptions about how these HRA methods conceive of and model human performance, is the goal of calculating the probability of a human error or erroneous action.

One primary shortfall with non-simulation HRA methods is the assumption that PSFs do not influence one another when, in fact, there is clear psychological evidence that PSFs frequently interact. For example, a limited amount of time to perform the task (i.e., time pressure) affects the person’s stress level when performing the task. Both time pressure and stress are commonly identified as separate PSFs in many HRA methods, which are used individually to directly modify the HEP in an additive fashion. Simply adding PSFs together simplifies the method, but it also eliminates any mathematical accounting for their potential interaction or influence upon one another. If these HRA methods are included in a simulation framework with this erroneous assumption that PSFs are independent and additive, the propagation of this error will lead to inaccurate HEP estimates.

This PSF example is just one of many issues with non-simulation based HRA methods. Others, including Swain [17] and Dougherty [25] have expounded on a range of issues with non-simulation based HRA methods that can greatly affect their ability to be effectively incorporated into simulation frameworks. Broadly speaking, issues include:

  • The accuracy of HRA’s HEP predictions has not been satisfactorily demonstrated, and

  • The time-dependence of human actions (i.e., dependency) is not effectively modeled for purposes of quantifying HEPs.

These issues, among others, need to be addressed with simulation based HRA methods in order for them to be effectively included into simulation frameworks.

The weaknesses in existing HRA methods are likely to be challenges in developing an approach to dynamic HRA. Recent work (for example, [26] and [27]) has focused on improving the accuracy and validity of HRA methods. Most of this work emphasizes the use of data collected in full-scope nuclear control room simulators such as the Human Systems Simulation Laboratory (HSSL) at INL (see [28] and [29]). Simulator studies may be invaluable stepping stones for building models required for simulation based HRA.

3 Simulation Based HRA

3.1 Introduction

In the face of any unresolved debate over different HRA methods, what advantage can be had by positing yet another approach? There exist developments—namely in human performance modeling—that do not fit the classification of existing HRA methods. Human performance modeling utilizes virtual scenarios, virtual environments, and virtual humans to mimic the performance of humans in actual scenarios and environments. What sets this form of HRA apart is that it provides a dynamic basis for HRA modeling and quantification. Traditional HRA methods have featured largely static task analyses of operating events as the underlying basis of performance modeling. These methods have also relied on performance estimations mapped to similar previous performance derived through empirical data or expert opinion. Simulation based HRA differs from its antecedents in that it is a dynamic modeling system that reproduces human decisions and actions as the basis for its performance estimation. Simulation based HRA may also augment previous HRA methods by dynamically computing PSF levels to arrive at HEPs for any given point in time. More importantly, simulation based HRA may present the decision points that operators make while engaging with the plant. These decision points are crucial anchors to plant performance, and no plant model can claim to model performance accurately without accounting for the nuances of human operations that determine the evolution of events.

3.2 Non-HRA Human Performance Modeling

Meister [30] suggests that HRA filled an important void early in the evolution of human factors by centering on prediction. Much of classic human factors has centered on the collection of data on the interaction of humans with designed systems. The purpose of such data is to improve the design of the system, ultimately to optimize human performance in terms of criteria such as usability, efficiency, or safety. HRA has instead attempted to predict human performance, specifically human errors, that can occur in such human-machine interactions. The purpose of HRA is therefore not typically to improve the design of the system so much as to determine what factors impact the safe human operation of that system. Over time, HRA has been joined by another predictive tool, namely human performance modeling.

Human performance modeling is an umbrella term used to describe systems that simulate human decision making and actions. Human performance modeling is largely synonymous with cognitive simulation and artificial intelligence, although it has in practice applied to unified systems that attempt to account for a broad range of human cognition. In contrast, variants of cognitive simulation or artificial intelligence may focus on modeling specific cognitive mechanisms instead of providing integrated models of multiple cognitive mechanisms. This distinction is analogous to the differences found in hardware component vs. system models, respectively. Young [31] suggests that human performance models vary on a number of dimensions, including:

  • The psychological theories underpinning the modeling,

  • The complexity of the human activity,

  • Models vs. applied simulations, and

  • The use of actual vs. conjectured human behavior.

There are numerous human performance modeling systems available. For example, [32] reviews human performance modeling systems that they have been applied to aerospace at the National Aeronautics and Space Administration (NASA), including the following systems:

  • Adaptive Control of Thought-Rational (ACT-R 5.0),

  • Air Man-machine Integration Design and Analysis System (Air MIDAS), which is a variant of MIDAS,

  • Distributed Operator Model Architecture (D-OMAR), and

  • Attention-Situation Awareness (A-SA) systems.

A recent review by Pew [33] for the golden anniversary issue of the journal Human Factors chronicles other human performance modeling systems, including:

  • Various versions of the Micro Saint task modeling system,

  • The General Problem Solver,

  • The State, Operator, and Result (SOAR or Soar) system,

  • The Goals, Operators, Methods, and Selection Rules (GOMS) approach,

  • The Executive-Process Interactive Control (EPIC) system,

  • ACT-R, and

  • MIDAS.

Recalling a general distinction between models and simulations, all of these systems offer models of cognition, but only Micro Saint, Soar, ACT-R, MIDAS, and EPIC are fully implemented simulation systems.

As noted, a defining characteristic of human performance modeling systems is that they mimic human decision making. Russell and Norvig [34] identify two general types of decision making used in human performance modeling systems. The first, historically speaking, is the deductive artificial intelligence approach, which consists of software systems that make simple deductive conclusions given coded representations. Two famous implementations include systems to prove logical theorems such as the General Problem Solver and logical programming languages such as PROLOG. The second logical reasoning type is the inductive system. Such a system, commonly called a production system, is capable of inferring from given contextual representations to produce new representations. The human performance modeling systems already described in [32, 33] mostly fit within this latter type of decision making. The advantages of inductive over deductive systems are striking: inductive systems can learn given minimal information, whereas deductive systems must avail preprogrammed information. Both, nonetheless, have their uses: the deductive General Problem Solver, for example, is quite effective at solving mathematical theorems, a domain that is certainly cognitive yet often falls outside the capacity of human cognition. The inductive logic production systems such as ACT-R, Soar, and Micro Saint, are more humanlike in their approach, making them suitable for simulating human performance realistically.

3.3 HRA and Human Performance Modeling

Gore and Smith [35] point out that despite a common focus on human performance, HRA and human performance modeling have not been well integrated. Human performance modeling systems have not been used to model those human behavioral contexts that lead to human error, nor to predict the rates of unsuccessful human performance. Yet, such an extension of human performance modeling is a logical bridge to HRA. Infusing HRA concepts like human error and HEPs into human performance modeling increases the utility of such systems.

Importantly for the present purposes, human performance modeling takes HRA out of the static models that are the mainstay of current Level 1 PRA applications.Footnote 1 While current HRA methods have proven robust in their application to Level 1 PRA, the methods are optimized for heavily proceduralized activities within the control room. Level 2 and 3 PRA require analyses of less proceduralized activities involving the dynamic interplay of control room and balance-of-plant and responder personnel. Current HRA methods are, with few exceptions, not validated for such applications. Human performance modeling affords the opportunity to extend current HRA approaches to novel domains by simulating Level 2 and 3 scenarios and the human activities within those scenarios.

There have been efforts to implement human performance modeling for HRA:

  • A framework for using NASA’s MIDAS system for HRA has been laid out [36] but has not been implemented to date.

  • ACT-R has been extended to model errors in the Human Error Modeling Architecture (HEMA) [37] in research funded by the Office of Naval Research. While the conceptual design was published in 2005, an implementation of the system has not been made public to date.

  • A production system inspired by ACT-R and Soar has been developed in conjunction with the University of Oldenburg to model errors by pilots and drivers [38]. This system helps identify sources of errors but does not predict their frequency.

  • A demonstration model in Micro Saint has mapped workload from the NASA Taskload Index (TLX) to the ATHEANA and SPAR-H HRA methods [39]. This research stops short of using the methods’ performance shaping factors to quantify human error. Its primary purpose in its current inception is to provide a mapping of existing workload simulation data to a format that is compatible with HRA methods.

  • The Accident Dynamics Simulator-Information Decision and Action in Crew (ADS-IDAC) system [40] was developed specifically for HRA applications, tying together a cognitive model, a decision making engine, performance shaping factors, and a dynamic event simulator. This implementation was further extended in [41, 42] to include a crew response model for emergency operations and severe accidents in nuclear power plants.

3.4 HRA and Human Performance Modeling of Severe Accidents

While other human performance modeling systems have achieved an otherwise adequate level of maturation in their domains, only the ADS-IDAC system requires minimal extensions to be used for HRA. Extending other human performance modeling systems to include HRA would require costly and time-consuming extensions of those preliminary efforts in [36–39].

In fact, ADS-IDAC recently was integrated with Methods for Estimation of Leakages and Consequences of Releases (MELCOR) code to evaluate a station blackout (SBO) at a pressurized water reactor (PWR) [43]. The project encountered significant technical difficulties, including challenges in the integration of ADS-IDAC with MELCOR. Instead of fully integrating the two programs, the authors suggest using an external script to jointly execute the two programs and manage interfacing data. RISMC’s toolkit is well-suited to this approach and might be a solution to the technical difficulties encountered previously.

However, coding ADS-IDAC scenarios requires significant resources. As with other HRA methods, ADS-IDAC was developed for Level 1 analysis. As such, the method is built around written procedures, with every procedure step and sub-step explicitly coded. Scenarios without written procedures are coded using “mental procedures” that can be activated when certain parameters or conditions are met. The level of detail in this model (and the associated resources required to implement it) may be too specific for the HRA modeling desired in the RISMC framework.

In any case, ADS-IDAC offers the most mature HRA-based human performance modeling currently developed. This model can be seen as the starting point for incorporating human actions into dynamic, simulation based risk assessment.

4 Conclusions

4.1 Selection of an HRA Approach for RISMC

For infrequent occurrences, including incidents at power plants, there is often inadequate operations experience to provide data-based quantification of human performance in HRA. Utilities, researchers, and regulators who wish to determine the risk significance of such past events retrospectively will utilize HRA estimation methods to the extent that they encompass the PSFs and scenarios at play in the event. However, because of the scarcity of available data, it is often necessary to utilize expert estimation techniques, which have historically been fraught with poor inter-analyst reliability [44].

Human performance modeling avoids the shortcomings of applying an HRA quantification method in a poorly suited domain or utilizing expert opinion to arrive at the human contribution to the risk of an event. Instead, by scripting a scenario that closely matches the past event, it is possible to generate simulation runs with virtual personnel to arrive at an estimate of the frequency with which human performance elevated the risk of the scenario. This approach increases the veracity of risk estimation.

Equally promising, so-called unexampled events, particularly severe accident scenarios, stand to benefit from human performance modeling by allowing virtual operators to engage in the evolution of events and provide a range of decisions and actions that might impact plant response. This form of simulation based HRA is a crucial evolution of risk analysis for the plant and one that can only be accomplished by coupling virtual operator models with advanced plant simulations. This problem set is the challenge of RISMC and presents an important opportunity to advance both HRA and the state of plant models.

4.2 Next Steps: Severe Accident Modeling and Need for Simulation Based HRA

In order to develop an approach for simulation based human reliability modeling, several questions need to be addressed:

  • What level of detail should be modeled? Is it appropriate to model operator cognition and every operator action, as in ADS-IDAC, or is a more high-level model sufficient? Perhaps a hybrid approach (e.g. the course-grain and fine-grain model proposed in [20]) should be adopted. This question is particularly relevant in Level 2 and Level 3 analysis, as these events are much less familiar to operators and procedures are not available for many of these scenarios. As we move away from prescribed human interactions with the plant, step-by-step analysis of operator actions becomes more difficult and more speculative.

  • What PSFs must be considered in simulation based HRA? PSFs are unlikely to remain constant throughout a scenario, and interactions between performance shaping factors must be addressed.

  • If an existing HRA method (or methods) is used, which method best suits the RISMC toolkit? SPAR-H was selected for RISMC’s first, simple HRA model [6]; perhaps this is a reasonable choice going forward. ADS-IDAC features nearly 60 PSFs [40]; yet development efforts may be best spent validating a subset of these PSFs like the eight found in SPAR-H [7].

  • When and how can empirical data and simulator studies be used to support dynamic HRA? Existing sources such as previous simulator study data may be useful, and INL’s HSSL provides a platform for collecting further data if desired.

Most of these questions are tied to the tension between a highly realistic but resource-intensive model and a model that is easy to implement and modify but perhaps too simplistic. The targeted balance between these two ends is complicated by the uncertainty surrounding human performance that has plagued HRA since its inception. In the next phase of this project, we will attempt to address these concerns and recommend an optimal approach for RISMC HRA.

5 Disclaimer

This work of authorship was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government, nor any agency thereof, nor any of their employees makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately-owned rights. Idaho National Laboratory is a multi-program laboratory operated by Battelle Energy Alliance LLC, for the United States Department of Energy under Contract DE-AC07-05ID14517.