Background

The implementation of Integrated Computational Materials Engineering (ICME) offers potential for significant benefits in all aspects of aerospace materials and processes engineering — including materials design and development, process modeling, and prediction of material behavior. There is potential to significantly reduce cost, time and risk for new materials and process insertion, to optimize material processing and properties for specific applications, and to computationally integrate materials and processing with other engineering disciplines [1]. Considerable effort has been expended on model development over the past three decades, and while many of these computational models are now in use, significant challenges to comprehensive development and implementation of ICME remain [2].

One of the key areas identified for future effort is the verification and validation (V&V) of ICME methods and models [2, 3] — including the need to develop guidelines and standards for this critical activity. The Air Force Research Laboratory sponsored an assessment of V&V activities in 2010 and 2011, with the objective of developing a recommended approach and practice for ICME V&V. This assessment included a review of relevant V&V activities in other engineering disciplines, and the development of a proposed general approach and guiding philosophy suitable for ICME. During this assessment, it became clear that some specific, pragmatic guidance on how to assess and plan ICME V&V activities would be useful, possibly even essential, to aid materials and process engineering practitioners. Several “practitioner aids” were developed including planning and execution checklists with instructions and examples, a recommended approach for Tool Maturity Level assessment, and an approach to assess “risk vs. consequences” of ICME application [4, 5]. These references are appended as “ Additional files 1 and 2”. This assessment and the practitioner aids were intended specifically for aerospace ICME applications, and were designed to be compatible with other gated review processes for technology and product development commonly used in the aerospace industry.

The need for ICME V&V

Engineering disciplines, such as Fluid and Solid Mechanics, have developed and validated mathematical and computational models and frameworks of greater maturity compared to those available today for ICME practitioners. A systematic, rigorous, and disciplined approach to verify and validate computational models and methods was deemed essential by these other disciples, to confirm the accuracy of model predictions. These disciplines have consequently pursued community-wide efforts to establish model verification and validation guidelines [6, 7]. Notably, representatives of these disciplines have asserted that the development and application of their guidelines were of paramount importance for building trust among both technical customers (e.g., materials application engineers and product design engineers) and managerial decision makers (e.g., product managers and ICME project sponsors).

The motivation and rationale for ICME verification and validation differ little from that of these other disciplines. However, owing to its relative immaturity, ICME has only recently begun to gain acceptance as a powerful method for enhancing materials analysis, development, and implementation. The application of verification and validation will accelerate the maturation of ICME by sharpening focus on the critical needs of its customers and associated decision-makers. V&V will help ICME practitioners to more clearly define of these needs and pursue improved model development trajectories, illuminated up-front by ICME accuracy and uncertainty requirements. By determining these metrics, the ICME community and customers alike will have quantitative information describing the maturity level of ICME technologies, promoting further acceptance and application benefit.

The importance and future value of ICME to the materials science and engineering community is broadly recognized, and is now the focus of the “Materials Genome Initiative” recently announced by the White House Office of Science and Technology Policy [8]. Hence, the development and implementation of a standard approach for ICME V&V is especially timely, even urgent, as increased focus, investment, and expectations are directed toward ICME.

Methods

The approach taken to develop this recommended practice followed three guiding tenets:

  1. 1.

    Utilize current, recognized practices for V&V as the basis

  2. 2.

    Facilitate alignment with established, gated review practices for product and technology development.

  3. 3.

    Provide simple, useful guidance and tools to aid practitioners in planning and assessing V&V activities and results.

The basis for ICME V&V

We have found that the ASME Guide 10–2006, “Guide for Verification and Validation in Computational Solid Mechanics” [7] provides a sound basis for guiding ICME V&V activities, and have subsequently used and referenced it extensively. It should be noted that the development of this document required extensive effort by a number of participants over a period of many years. The Guide provides an excellent set of definitions and guidelines, and is applicable to a broad range of computational disciplines and applications.

It is useful to ensure common understanding of the terms “verification” and “validation” for this application [7]:

· Verification: the process of determining that a computational model accurately represents the underlying mathematical model and its solution.

· Validation: the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model.

Verification, in this context, means ensuring that computer code performs as intended and equations are solved correctly. Validation, in this context, means assessing the extent to which a model represents the “reality of interest,” over the range of intended application. Key guidance from the ASME V&V Guide that we regard as especially important V&V of ICME methods and models is:

· Verification must precede validation; and when used, calibration must precede validation.

· The need for validation and the specific computational accuracy requirements depend on intended use, and should be considered as part of the V&V activities.

· Validation of any complex system should be pursued hierarchically.

· Simulation results and experimental data must be generated independently and have assessment of uncertainty to be meaningful.

There is extensive discussion and illustration of the recommended processes for verification and validation, including these specific items, in ASME V&V Guide 2006, and by Thacker [9]. In addition, these references stress the importance of Uncertainty Quantification (UQ) as an essential focus area for V&V. While UQ is expected to be especially critical for ICME V&V and the subject of future publications, it is not addressed in any detail in this document.

Alignment of ICME V&V with product and technology development processes

Successful, broad implementation of ICME in the future will span an enormous range of applications. ICME encompasses all areas of materials and process engineering, from early development of technology through product support in the field. The application of ICME may involve simple trending or guidance for development, or activities as complex and critical as life prediction in safety-critical systems. The consequences of using ICME for decision-making will vary accordingly. Further, the guiding philosophy proposed for ICME V&V should also align with established product development processes, such as Integrated Product Development (IPD) and the US DoD’s Defense Acquisition Milestones [10], and technology development processes such as Technology and Manufacturing Readiness Levels — TRL and MRL, respectively [10, 11]. These are all “gated review processes” — meaning that there are rigorous criteria for assessing completion of a particular gate, and that progressive stages imply more substantial commitment and investment. These processes, or similar ones, have been adopted by various companies and agencies, so they are broadly recognized and widely used. The level of maturity of a particular technology — for a specific application — is now readily communicated by simply stating its current “TRL status.”

Similarly, ICME methods and models would benefit from a Tool Maturity Level (TML) assessment process, which could be used to guide development and application of ICME methods and models analogous to the TRL ranking process. If accepted and broadly used, as is the TRL process, a Tool Maturity Level would readily convey the state of maturity of a particular ICME model or method, and offer objective guidance on where in the TRL or IPD process its use would be appropriate. Finally, TML criteria could, and should, be considered in light of the potential decisions or consequences of application. These assessments are critical steps in determining the level of ICME V&V needed for a specific application, and are instrumental in planning and executing V&V activity.

Guidelines and practitioner aids for ICME V&V

The philosophy and basis for the recommended approach to ICME V&V are relatively easy to describe and reference — but somewhat more difficult to implement in a consistent and sustained manner. Consequently, some simple tools for V&V planning and assessment that enable practitioners to get started on ICME V&V immediately are essential. The most important tools for this are ICME V&V System-Level and Model Checklists, a Tool Maturity Level (TML) assessment guide, and an approach for assessment of specific application risks. Initial versions of these tools have been drafted in spreadsheet format, along with a simple set of process steps to guide practitioners in their use [4, 5]. The recommended process for using these tools, in its simplest form, is presented in Figure 1.

Figure 1
figure 1

Flow chart of simplified ICME V&V planning and assessment process.

Full size image

This is, of course, a simplistic representation of an effective V&V planning and execution process, but illustrates the logical sequence of activities needed, and that in fact iteration may be required to arrive at an acceptable ICME V&V plan for a specific application, including any risk mitigation activities that may be indicated. A more detailed process description is presented in the referenced reports [4, 5].

These tools will facilitate a consistent approach to ICME V&V, and help ICME developers and practitioners bridge the critical step from general guidelines and procedures to their specific applications. We expect that these tools will be refined and improved significantly with time and experience.

V&V for special causes

There is significant potential for ICME to improve our understanding of new materials and processes, and reduce risk for early applications, if we employ a rigorous V&V approach and utilize the TML and risk assessment approaches outlined here to consider and assess the possibility of infrequent or rare events. These are sometimes described as “special causes,” especially when referring to product failure events.

ICME practitioners and customers gain confidence when V&V efforts succeed in demonstrating that a modeling system accurately describes the “physical reality” of interest, quantifies modeling uncertainty, and clearly defines the range of applicability. Sometimes, ICME goals are modest, for example when the system is intended to screen alloy compositions or process trajectories in support of detailed research planning. For such low-risk cases, modeling and subsequent V&V can safely adopt a narrow focus on the most dominant and expected processing and behavioral mechanisms. Correspondingly, defining customer needs, the reality-of-interest, accuracy/uncertainty requirements, and consequent tool maturity level is straightforward. Conversely, the difficulty of defining and confirming acceptable ICME system capability increases significantly when goals are both stringent and complex and the required tool maturity level is necessarily high in order to address the later stages of the TRL process. Examples of this latter category include ICME systems that support production qualification of a new process or the determination of minimum properties of a new material for use by life management engineers.

A late TRL material system that involves “special causes,” a subset of the latter high-risk category, poses a significant challenge to ICME and V&V — much as it has frustrated today’s empirical, data-driven materials engineering approaches. Special causes produce anomalous process defects, microstructures, and/or material behavior that are difficult to detect via experiment or test because their frequency of occurrence is typically low and/or their full impact is unknowable a’priori based solely on experience. As a historical example, consider the introduction of hot isostatically pressed (HIP) powder metallurgy (PM) superalloys in the 1970s. At the time it was well known that these As-HIP PM alloys were defect sensitive and that there were a multiplicity of different types of low-cycle-fatigue (LCF) initiation sites among failed test bars. However, one defect type, produced by organic powder contaminants whose gasification/oxidation weakened interparticle bonds, ultimately led to abandonment of As-HIP PM processing. Despite an unprecedented number of mechanical tests, the Achilles heel of As-HIP superalloy processing wasn’t fully revealed until an engine failure gave the ultimate verdict.

The difficulty posed by “special causes” upon ICME and V&V is significant, but these anomalies also offer a great opportunity for advancing the benefit of ICME via V&V. Unlike traditional data-driven methods, ICME and V&V can adopt a proactive approach to analyzing and modeling “special causes” including both anomalous processing artifacts and material behavioral issues. The role of V&V is to:

· Focus up-front on both the explicit and implied requirements of the customer.

· Exhaustively examine the conceptual model to assure that all relevant physics is included — both mechanisms for nominal behavior and exceptional mechanisms associated with “special causes”.

· Exercise the ICME modeling system to evaluate process and material behavior outside the nominal processing window to assess process robustness and evaluate the influence of processing discrepancies.

· Proactively analyze the likelihood and effect of “special causes” through detailed modeling or approximate methods.

· Exercise risk management to identify, assess, and abate material and processing “special causes” outside the purview of the ICME team owing to technical or programmatic constraints.

These concepts are mentioned subsequently in the ensuing sections of this paper that discuss the details of the proposed V&V approach. We now offer a simple hypothetical example to make some aspects of the outlined approach more concrete.

A gas turbine engine manufacturer has undertaken development of a new product with a very aggressive schedule. The stringent schedule is deemed acceptable because, although the engine is significantly larger than its predecessor, it will use no new materials. However, the turbine design manager questions whether established design minimum properties for the cast and wrought disk alloy can be achieved in the larger stage 1 turbine disk. The manager conveys his concerns to the ICME team and requests an analysis of disk heat treatment and a projection of mechanical properties because the larger forgings won’t be available for property testing for another year. The team drafts a conceptual model focused on heat treatment, precipitation, and mechanical property modeling. However, during V&V review of the conceptual modeling plan, one ICME engineer notes that the larger disk-forging billet will require a larger cast ingot outside of prior experience. The engineer expresses concern that solidification of an ingot this size would influence dendrite-arm-spacing, grain structure, the scale of interdendritic segregation, and even could lead to freckle segregation, a “special cause” condition. Also during V&V risk assessment, the heat treatment modeler questioned the validity of assumed quench heat transfer coefficients because the larger disk requires a larger quench tank with a different agitation system. The ICME team then conducted analyses to assess the likelihood of ingot freckling and estimate quench-process robustness. Based on the team’s findings they extended their analysis, worked with the suppliers to confirm and resolve these issues, and thereby avoided a significant schedule disruption during early production stages of the program.

In conclusion, although focused up-front consideration and ICME predictive analysis of “special causes” will increase the likelihood of early detection and resolution of some, a few will undoubtedly escape early identification. However, even for these latter cases, ICME and V&V can be applied to more quickly assess and resolve such surprises when they do occur.

ICME V&V checklists

Although the Verification and Validation Checklists below were developed as practitioner aids for aerospace materials researchers we believe that they are applicable within the broader materials community. The checklists are intended to provide guidance to ICME development teams, particularly those with limited experience with V&V. In formulating the checklists, an expansive view of V&V has been adopted that extends beyond the technical strategies and techniques necessary to perform V&V. If a major goal of V&V is to build trust in ICME system predictions, it is paramount for a V&V team to understand the intended use of the system as well as the needs and expectations of those who rely on ICME predictions — the technical customers and managerial decision-makers. For this reason, non-technical activities have been included in the checklists, as described below.

Introduction and precepts

The V&V precepts, underlying the V&V checklists, challenge some common shortcomings of materials modeling and include the following:

Understanding customer needs

· An ICME model should serve a real customer need by supporting material and process design, application, and support decisions.

· ICME predictions must have sufficient accuracy to meet this need. For low risk decisions, even a model that provides proper trends may be useful.

· The customer/decision-maker must trust the modeling results. Decision-makers trust sources of information and analysis methods that are widely accepted and have proven reliable. V&V should substantiate that ICME predictions provide equal or better confidence than historical precedents.

· Dialog with the customer needs to continue throughout the ICME V&V process to address identified issues and risks that bear on the implied as well as explicit customer needs.

ICME V&V is a continuous process

· ICME V&V should begin at program start and continue until program completion. By doing so, V&V teams define customer needs, application scenarios, and accuracy requirements early. Armed with these inputs, the ICME team can better define the ICME model, identify modeling risks, identify validation plans, and design validation uncertainty quantification analyses. This precept counters the too common notion that V&V should be considered only after ICME model development has been completed. Indeed, too many traditional materials modeling programs iteratively carryout model refinement and never attempt formal validation.

V&V activities promote team-wide communications

· Productive ICME V&V demands integrated product development (IPD) teamwork and communications among participants having diverse backgrounds and expertise. ICME V&V practitioners inform and engage other stakeholders (material specialists, application engineers, product engineers, and other customers) about modeling goals, approaches, limitations, and risks, including those associated with “special causes”.

V&V risk assessment and uncertainty quantification improves model performance

· V&V encourages early identification of unintended side-effects from inappropriate modeling assumptions, formulation of more realistic boundary conditions, assessment of alternative modeling approaches, recognition of shortcomings with model input data, and resolution of implementation issues.

· Focused attention of ICME risks fosters the early identification of material processing and behavioral mechanisms that allow higher fidelity predictions that more closely simulate the reality-of-interest. Such inclusiveness enables prediction of off-nominal materials and processes including some “special cause” events.

· Model improvement requires that the ICME development team quantify and understand model uncertainty and errors; this includes: i.) Model parameter sensitivity analysis, ii.) Quantification of model input errors and variation, iii.) Analysis of uncertainty propagation throughout the modeling system, and iv.) Uncertainty analysis to quantify model fidelity during model validation.

Description of the ICME V&V checklists

V&V ICME system and model level checklists were constructed to provide a “quick start” listing of traditional V&V activities that describes “what” should be considered and addressed during V&V activities without mandating “how” these activities should be carried out. As such, the V&V checklists are not prescriptive in that each ICME team can determine which elements of a checklist are relevant and helpful to their project. The checklists were designed to be straightforward and simple to fill out and maintain. These V&V checklists will evolve as ICME teams gain greater V&V experience.

The ASME V&V guide [7] advocates that V&V should be approached hierarchically within a modeling system (i.e., from components, through subassemblies, to full systems). This recommendation promotes a systematic approach that helps identify individual model inadequacies and cumulatively builds confidence in the overall modeling system. The proposed ICME V&V guidelines agree with this concept but also acknowledge that materials systems usually involve interacting model networks that are more complex than the tree-like morphology common to the mechanical disciplines. A simplified (and incomplete) ICME example is shown in Figure 2. It remains important to decompose such complex ICME modeling systems into meaningful constituent subsystems. Given the benefits of hierarchical analysis, at least for large ICME systems, two checklist templates were developed — one for individual models and a second for the ICME system and application. The ICME checklists are summarized in Table 1 that lists both the categories and associated checklist items; this table separates system from model elements. Notice that the system checklist deals primarily with ICME customer needs, business case, and systemic considerations and risks, whereas, the modeling V&V checklist focuses upon model development, technical risks, verification, and validation. The model checklist attempts to adhere to the process flow advocated by the ASME V&V guideline [7]. The complete checklists are available in Microsoft Excel™ format.

Figure 2
figure 2

Interaction between process, microstructure and properties within an ICME system.

Full size image
Table 1 Outline of the V&V Checklists
Full size table

Tool maturity level (TML) and application risk assessment

The TML Assessment Guide was intended to facilitate assessment of maturity and capability relative to intended ICME applications, and to provide a useful tool for guiding and assessing ICME V&V activities during model development or refinement when integrated with the ICME V&V Checklists.

Currently, there is no simple, standard, gated review process in broad use for assessment and communication of the maturity level of an analytical model or tool — at least not in the same sense as the TRL process. Sandia National Laboratories and NASA have published approaches and standards for predictive capability and maturity of computational models and simulations [12, 13]. Both are excellent references and guidelines. The Sandia report describes levels from zero to three in maturity, and contains very specific criteria descriptions for various assessment elements. NASA-STD-7009 describes levels from zero to four (5 levels), and has more general descriptions which would be adaptable for various system level applications. Interestingly, the NASA standard includes practitioner (personnel) capability and training as one of the major assessment elements.

What we propose for ICME use is based on a simple five-level maturity assessment as presented by Morris [14], at the 8th International HCF Conference in 2003. This approach, adapted for ICME, is illustrated in Table 2.

Table 2 Tool Maturity Level (TML) Description for ICME (Adapted from Morris,[14])
Full size table

The concept is simple, useful for ICME V&V process development, and provides an approach to align ICME V&V with technology and product development requirements. The assessment elements are briefly described in Table 3. The assessment elements and gate criteria, while developed specifically for ICME applications, are generally consistent with those in both the Sandia report and the NASA standard previously referenced.

Table 3 Tool Maturity Level (TML) Assessment Elements
Full size table

Determining what tool maturity level is required for an ICME model or method — and consequently the level and fidelity of verification and validation of a particular analytical tool — is of course dependent upon the specific application. The maturity requirements will very likely increase as the region of ICME application progressed in the technology or product development process. This is not to say that “high TML” ICME tools are not needed or desired in early stages of technology or product development – but rather that the later stages require higher TML levels because the potential consequences of application generally become more severe.

Since there will likely be many specific ICME applications in the future, developing a very prescriptive guideline regarding TML requirements is neither likely nor desirable. Consequently, it is useful to consider a supplemental means of assessing “risk vs. consequences” of ICME application, and using the outcome to determine whether the V&V status of a specific ICME model is sufficient. An excellent example of a “Risk vs. Consequence” table was developed by NASA [15] and is reproduced in Figure 3. The associated text boxes and arrows indicate how it relates to ICME model and V&V considerations, and where in the TRL or IPD process a specific ICME tool might be applied.

Figure 3
figure 3

Alternative risk analysis approach to assess ICME tool maturity[15].

Full size image

This tool is used in conjunction with the Checklists and TML assessment for ICME development and validation planning. A “risk vs. consequences” assessment is intended to provoke an assessment of deficiencies or risks that should be addressed in conjunction with specific ICME applications. This allows ICME tool developers to objectively assess their potential impact and develop risk mitigation actions as warranted.

Our long-term vision is that the TML Assessment becomes a “gated process,” where expectations are that all prior TML criteria are met before a particular analytical model or tool can progress to the next level but where flexibility is maintained to permit tailoring the assessment to the many, diverse applications which ICME will surely encounter in the future.

Results and Discussion

Verification and validation examples are needed to help practitioners accelerate and strengthen the application of V&V within ICME development programs. Of course, the best and most relevant examples will ultimately be drawn from successful ICME projects involving an integrated system of models. However for now, a simple ICME example is offered that illustrates some of the V&V concepts and checklist elements. The example is based upon research by one of the authors involving implementation, verification, and validation of a single, publically available precipitation model for 6000 series aluminum alloys.

Customer Needs (Checklist S1) — An ICME development team identified the need to establish a model to describe precipitation during the aging of aluminum A6082 thin-wall extrusions; it would be used provisionally within a larger integrated multi-model ICME system. The customer for the precipitation-modeling project required rapid identification and implementation of a model capable of simulating A6082 hardness trends as a function of heat treatment process variables. The model was expected to benefit the overall ICME development effort by providing guidance for development of a final, more comprehensive precipitation model and allowing earlier ICME system studies. Although model accuracy requirements were modest, the customer and development team agreed that verification and validation were required to assure that the model produced reasonable trends and to provide data that could be used in subsequent verification of the more comprehensive final model.

The precipitation model and associated V&V

The ICME development team identified a model, developed by Myhr [16, 17], for implementation as the provisional precipitation model for the A6082 alloy. This model is physically based and applies classical precipitate nucleation and growth theory in combination with Friedel’s precipitate strengthening theory. The model focuses on the dominant β” strengthening precipitate, uses pragmatic thermodynamic simplification, and relies on experimentation to calibrate the nucleation rate equation. The Myhr papers describe this precipitation model in full detail along with associated validation experiments and results. Although Myhr et. al. verified and validated the precipitation model, the case study model required additional verification of its MatLab™ implementation and A6082 validation experiments because this alloy’s composition and specific processing route could fall outside the range addressed by Myhr’s research.

This case study was carried out prior to development of the V&V checklists; however retrospectively, the implementation of the Myhr precipitation model for A6082 was performed in accordance with key elements of the AIAA [6] and ASME [7] guidelines as well as the V&V checklists. The ensuing case study description is organized using these guidelines and reference the V&V checklist where appropriate.

The Sargent diagram [6, 18], Figure 4, provides an overview of the V&V approach executed within the case study. The study followed the three successive steps shown in the diagram, including: i.) Confirmation of the Myhr model prior to implementation, ii.) Verification that the mathematical model was properly implemented within MatLab™, and iii.) Validation assessing model fidelity and uncertainty when applied to aging of A6082.

Figure 4
figure 4

Sargent diagram showing overview of case study V&V activities[6, 18].

Full size image

Model Confirmation

Assessment of the precipitation model was made easier because Myhr had published a comprehensive description of the model physics and mathematical implementation within a peer reviewed journal [16, 17]. The precipitation model includes a heterogeneous nucleation law, a relationship the Gibbs free energy, a linear precipitate growth equation, the Gibbs-Thompson equation for precipitate solubility, and a continuity equation governing the evolution of the precipitate particle size distribution. The model also includes strengthening relationships based on Friedel’s formulation to derive alloy yield strength and hardness values based on the size distribution of the strengthening precipitates. While the confirmation exercise (Checklist M1) concluded that these are all necessary elements, it also identified several potential weaknesses in the model formulation:

· Thermodynamics: The model uses simplified, approximate thermodynamic relationships rather than thermodynamic software such as ThermoCalc™.

· Nucleation Rate: The model does not account for precipitate incubation or the variation in potency among heterogeneous nucleation sites.

· Growth Rate: The model ignores the non-linearity of compositional profile in the matrix and impingement effects.

These issues could influence accuracy and miss trends associated with differing cooling rates following solutionizing, particularly for rates that are much slower than those studied by the original model developers. Although the assessment team concluded that rectifying these issues would exceed the scope of the study and lose precious time, it decided to track these issues within the project risk management plan (Checklist S3).

Verification of the MatLab™ Computational Model

Verification (Checklist M3) for the A6082 precipitation model included three steps that involved checking the MatLab™ code to identify and resolve errors; comparing simulation results against a benchmark computation; and examining model output for anomalies. The original Myhr papers described the computational implementation thoroughly, thereby reducing debugging to little more than correcting several simple code syntax and typographical errors.

The benchmark verification entailed comparing output for the MatLab™ implementation against results reported in Myhr [16]. The comparison included checking predictions of nucleation rate, particle number density, mean particle radius, and the particle size distribution at discrete times during isothermal aging. For each output characteristic, output from the MatLab™ implementation matched the published Myhr results quite well. A comparison of predicted precipitate particle size distributions is shown in Figure 5. The two sets of modeling results nearly overlay each other except near the peak of the distribution; this discrepancy was attributed to likely small differences in the time and/or radius (particle size) increments used for numerical integration.

Figure 5
figure 5

Comparison of particle size distributions calculated by Myhr[16](benchmark) and the current MatLab™ implementation following 10 hours age at 180°C.

Full size image

The benchmark verification demonstrated that the model was properly implemented within MatLab™. However, subsequent application of the model for A6082, using a modified thermodynamic relationship provided by Myhr [17] revealed an unexpected anomalous secondary spike in the predicted particle size distribution. This peak, shown in Figure 6, was transient, forming and disappearing within a relatively brief time window. The presence of the anomalous peak required additional verification that involved determining whether it reflected a numerical instability, a discontinuity in the revised thermodynamic formulation, or a spurious nucleation event.

Figure 6
figure 6

A6082 precipitate size distribution exhibited a transient anomalous peak following a 2 hour aging heat treatment at 180°C.

Full size image

Study of the anomaly, involving changing the integration time and spatial increments along with use of the earlier thermodynamic relationship, failed to eliminate the anomalous peak. However, careful examination of mass flows among cells in the vicinity of the peak and inspection of the local cell mass balance, indicated that the anomalous peak was a natural consequence of the nucleation formulation. Specifically, the Myhr model assumes that all nuclei forming during a time step have a fixed radius, which leads to an exaggerated increase in number density for that precipitate size. This conclusion was confirmed by “turning off” nucleation during the critical time window, which eliminated the secondary peak. Despite this finding, the team decided against attempting to fix the nucleation formulation (e.g., by dispersing the nuclei radius) because no data was available to guide or validate such an adjustment; and furthermore, the spurious peak had no significant effect on yield strength predictions, as verified via analysis using the ancillary strength model.

Validation of the A6082 Precipitation Model

The Myhr research team [16, 17] had rigorously validated the foundational model for a range of aluminum-magnesium-silicon alloys and select thermal cycles. This work included activities such as electron microscopy to measure precipitate sizes and number densities, and hardness testing to infer yield strength. Validation and uncertainty assessment of the A6082 MatLab implementation was significantly more modest and included both experimental and modeling activities. In overview, model validation consisted of comparing simulation of precipitation and subsequent hardening against experimentally determined aging curves that describe A6082 hardness as a function of aging time. These efforts, listed below, included activities described in sections M2 and M4 of the V&V checklist.

· Measurement of temperature variation during heat treatment and hardness measurement error;

· Heat treatment trials involving the aging of coupons for differing lengths of time;

· Replicated hardness measurements for each coupon coupled with subsequent calculation of hardness uncertainty;

· Simulation of precipitate size distributions and associated hardness values for A6082 coupons heat treated per the experimental plan; and

· Synthesis of a lower bound of simulation error via the Monte Carlo method using the results of a sensitivity analysis and uncertainty estimates for model inputs and parameters. The results were judged to represent a lower bound because not all sources of uncertainty were included in the analysis.

The results of the validation exercise and embedded uncertainty assessment are shown in Figure 7. The graph shows that the A6082 precipitation modeling results (in combination with the hardness model) agree well with the experimental result and both exhibit comparable levels of uncertainty for the alloy and heat treatment used in this study.

Figure 7
figure 7

Comparison of experimental and modeling validation results for the MatLab Implementation of the Myhr precipitation model.

Full size image

TML assessment and “risk vs. Consequences” example

As with the ICME V&V example described in the previous section, example applications of the Tool Maturity Level assessment guide, and associated risk vs. consequences assessments, are needed to help practitioners accelerate and strengthen the application of V&V within ICME development programs. Again, the best and most relevant examples will ultimately be drawn from successful ICME projects involving an integrated system of models targeted for specific applications. A hypothetical example based on the results of a recent Air Force program to develop standardized residual stress measurement and modeling techniques is provided below.

Consider a comprehensive ICME effort aimed at predicting bulk residual stresses in an aerospace component, which will include process-induced residual stresses from forging and heat treatment, and the redistribution of these stresses after final machining. The goal of this ICME program is to reduce the machining cost by predicting and managing any distortion within the desired final part envelop and to integrate the analytically predicted bulk residual stress effects into final component service-life predictions. Model development includes new modules as well as the extension of existing tools by incorporating new data related to the specific material and application. V&V plans have been developed in accordance with the Checklists with the intent to achieve TML-3 at completion.

TML-3 [4] represents a mature analytical tool or model — one that could significantly reduce or eliminate iterations in a material or process development program, and potentially reduce experimental testing or other analytical requirements. At TML-3, the model can be used directly for assessments or evaluations of derivative materials or processes, or deviations from known practices. While the accuracy or fidelity of the model may not be fully validated, the range of application should be well defined and documented. In addition, supporting data should represent the entire range of application, a User’s Guide should be fully developed and updated (from TML-2 level), version control for software implemented, and significant sub-model validation completed. Finally, at TML-3, a Risk vs. Consequences assessment is highly recommended prior to any significant application.

In this hypothetical example, it is uncertain whether TML-3 will be totally achieved, and what impact that might have on realizing all the program and project goals. Further, it is uncertain whether TRL-3 is an adequate maturity level to achieve the most ambitious project goal: use of the predicted residual stress fields in the component life predictions for fatigue and damage tolerance. Consequently, the developers and project team generated a high-level flow diagram from their System and Model level checklist information, to help assess the critical elements for V&V, as illustrated in Figure 8.

Figure 8
figure 8

Simple flow map for residual stress hypothetical example.

Full size image

The project and development team identified five specific items that appear to be significant risk areas that warrant additional consideration in the V&V plan for this application. These are indicated with numbered symbols in Figure 8. The team then constructed a working table to list the specific risks, estimated the likelihood of occurrence and the severity or potential impact to the project.

The developers and project team determined risk mitigation actions, where appropriate, and estimated the effect on reducing program risk and potential consequences:

  1. 1.

    Thermal process modeling must accurately predict temperatures and temperature transients during processing in order to predict resulting residual stress fields. The project team performed a parametric analysis where such predictions were evaluated for typical aerospace rotating components. This analysis found that determination of accurate heat transfer coefficients, especially for use during high transient thermal processes, was critical. Consequently, additional measurements of part temperature during processing and use of complementary methods to estimate heat transfer coefficients, were deemed necessary.

  2. 2.

    Measured residual stresses must reasonably validate predicted values for the intended purpose. Virtual experiments were performed with combined error sources using nominal magnitudes in order to estimate the current stress prediction error for the residual stress measurement technique. In performing the virtual experiments, emphasis was placed on simulating "worst-case" scenario error sources in order to understand the lower bounds of accuracy associated with the method. Based on the results of the virtual experiments the team planned additional stress-relaxation mechanical tests to ensure adequate constitutive data to support the modeling efforts. The team also decided to use multiple methods for residual stress measurement validation.

  3. 3.

    Variation in final microstructures must be within desired control limits for the intended application — to ensure that material properties and behavior meet design intent. The project team recognized this as a potential risk, but has experience with microstructure evolution modeling for this material and determined that the risk can be mitigated or eliminated with early production monitoring and controls.

  4. 4.

    Distortion during finish machining may result from redistribution of process-induced residual stresses. This may result from placement of the final part within the forging envelope, as was found in a recent case study for an aerospace aluminum forging [19]. For the rotating parts considered in this project, the team regarded this risk as low but required simulation of final part placement in the forging envelop, and assessment of machining sequence effects on distortion.

  5. 5.

    Fatigue and damage tolerance life predictions require high fidelity stress values for required accuracy. The project team was concerned that the accuracy of residual stress predictions may be suitable for use in defining forging shapes, reducing input weights, and predicting or controlling part distortion during machining, but not sufficiently accurate to use directly in life predictions. Further, potential errors in residual stress measurement accuracy may preclude or limit validation. The project team regarded this as a risk that was likely to occur based on similar studies [19], and that would have significant impact if it did occur. The team adjusted this specific project goal and decided to use the residual stresses for static assessments such as potential impact on rotor burst limits, rather than directly in Low Cycle Fatigue (LCF) and fracture mechanics life predictions. The team agreed to reassess at a future date, pending improved elements of the ICME modeling efforts, measurement techniques, or demonstrated ability to achieve TML-4 for this process.

The resulting list of risks, estimated likelihood of occurrence, estimated consequences, and a simple “red-yellow–green” assessment of importance are summarized in Figure 9.

Figure 9
figure 9

Example of risk identification, impact and mitigation estimates.

Full size image

Expected effects of the planned mitigation items are also shown in Figure 9, with changes indicated by the small arrows. The five risk items are plotted on the 5X5 Risk vs. Consequences Matrix in Figure 10, at the assessed levels for likelihood and consequence. This figure illustrates the effect of the planned mitigation actions, and improvements to the V&V plan for this ICME project. The assessment indicated that Items 1 and 3 could be improved from “yellow” to “green” with mitigation actions, and that Items 2 and 4 posed some risk but did not require any risk mitigation actions. Item 5 obviously posed a high risk which could be reduced with mitigation actions but would still warrant attention of the project and development team.

Figure 10
figure 10

Example use of risk matrix and effect of mitigation actions.

Full size image

This is just a hypothetical example, of course, but it shows how a project risk management tool can be used in conjunction with the ICME System Level and Model V&V Checklists, and the TML Assessment guide to assist ICME V&V planning, especially for developmental ICME models or methods, or when development program decisions will be ICME based.

Conclusions

ICME has the potential to greatly benefit the materials science and engineering communities, and to greatly enhance integration with other engineering disciplines. The potential benefits of ICME to reduce time, cost, or risk, and to enhance future “design” of materials and processes, are enormous. Verification and validation of models and methods poses a significant challenge to broad development, implementation, and acceptance of ICME. This is especially relevant where significant decisions will be ICME-based. That is, where decisions may affect technology or product development, legacy system sustainment actions, supply base decisions, or quality assessments.

The approach for ICME verification and validation described in this paper was developed based on a philosophy aligned with current well-established technology and product development processes. The guidelines are consistent with, and frequently reference, the well-established ASME V&V Guide 10–2006, developed by the computational solid mechanics community after many years of effort. Finally, the tools that were developed to aid ICME developers include an ICME V&V System-Level Checklist, a Model V&V Checklist, a TML Assessment Guide, and a risk matrix tool for project risk assessment and mitigation. Much effort has gone into making these tools broadly applicable, useful, simple, and flexible — and then integrating them to ensure compatibility. Instructions for their use, and initial examples of their application were developed.

The authors recognize that this is and should be an evolving effort. The tools and approach outlined here will be continuously improved and updated with increasing application experience.