Findings

Background and hypothesis

It is believed that broad development and implementation of Integrated Computational Materials Engineering (ICME) offers potential for significant benefits to all aspects of aerospace materials and processes engineering - in the form of greatly reduced time, cost and risk of technology development, validation, and sustainment. One major challenge for broad acceptance and implementation is that of verification and validation (V&V) of ICME models and methods. The US Air Force Research Laboratory (AFRL) sponsored efforts related to development of guidelines for ICME V&V. The approach taken to develop this recommended best practice followed three guiding tenets:

  1. 1.

    Utilize current, recognized practices for V&V as the basis. This was predominately the American Society of Mechanical Engineers (ASME) V&V Guide - 2006 [1].

  2. 2.

    Facilitate alignment with established, gated practices for product and technology development. These included the Technology and Manufacturing Readiness Level processes (TRL and MRL, respectively) and both the Integrated Product Development (IPD) and Defense Acquisition Guidebook processes for system or product development.

  3. 3.

    Provide simple, useful guidance and tools to aid practitioners in planning and assessing V&V activities and results for ICME development.

The rationale and approach for developing these guidelines and practitioner aides were described in a journal article in 2012 [2]. The initial versions of the recommended best practice, practitioner aides such as V&V checklists, a Tool Maturity Level (TML) assessment approach, and a recommended approach to assess risk vs. consequences of ICME application were appended as additional files accessible through the IMMI journal reference.

It was expected that the initial guidelines and practitioner aides would benefit from a revision cycle following some application experience. Several AFRL- and Defense Advanced Research Projects Agency (DARPA)-sponsored programs have since reviewed and/or applied the guidelines. Feedback from these programs and from individual experts was used to generate a comprehensive revision to the original guidelines and practitioner aides. This effort was performed under AFRL sponsorship of the Metals Affordability Initiative (MAI) program ‘GE-12: Verification and Validation (V&V) of ICME Models and Methods’ (Agreement Number: FA8650-08-2-5251).

Methods and approach

Feedback was solicited from numerous individuals engaged in ICME development programs representing aerospace propulsion companies and from within AFRL. Participants represented three programs where the guidelines and practitioner aides were used directly, in whole or in part. In these cases, technology and/or intended product applications were well defined, as was the intended role of the ICME elements under development. Responses were also solicited from selected individuals with prior ICME development and implementation experience. Feedback was collected over an 18-month period, from May 2012 through October 2013.

Feedback was generally consistent and focused on the need to simplify wording, reduce the number of line items in the practitioner aides, and clarify the recommended process flow for V&V planning and execution. The feedback was collectively reviewed, consolidated, and used to generate revisions to the ICME V&V guidelines, the recommended best practice, and the associated practitioner aides. The text document containing the revised guidelines and recommended best practice [3] and the associated Excel™ file containing the revised practitioner aides [4] were released by AFRL for public distribution in January 2014. These references are appended as Additional files 1 and 2.

Results

The general form of the guidelines and recommended best practice for ICME V&V planning and execution was preserved in the updated version, as itemized in the Background and hypothesis section, above. The recommended process flow chart, as revised based on the effort described above, is shown in Figure 1.

Figure 1
figure 1

Flow chart of the simplified ICME V&V planning and assessment process.

Full size image

The flow chart indicates the key steps in the process and where the practitioner aides may be helpful. The flow chart also indicates the expected iterative nature of developing an ICME V&V plan based on a specific technology or product application. A more detailed flow chart, and associated descriptions of the steps involved, is presented in the appended files [3,4].

Specific changes and improvements that were included can be summarized as follows:

  • System-Level and Model-Level Checklists: These practitioner aides were simplified in wording and number of line items. The supporting text was significantly revised and expanded to more clearly communicate intent and recommended process for use of this checklist.

  • TML assessment guide: This practitioner aide was revised by adding numerous supporting comments in the Excel™ comment tabs, adjusting some of the category headers based on feedback, and by adding some line items in the category boxes for selected TML levels.

  • Recommended Best Practice for ICME V&V: The two recommended process flow charts were revised. The simplified (‘3-step’) process flow chart was updated to illustrate the iterative and interactive nature recommended for key steps in the V&V planning process. The detailed process flow chart was revised to improve flow and reflect the expected actual sequence of activities in a V&V planning process. In addition, the step-by-step instructions were updated and recommended activities clarified.

  • Use of Phenomena Identification and Ranking Technique (PIRT) (note that often ‘Table’ is used instead of ‘Technique’ in this acronym): Instructions and an ICME-relevant example for the use of this technique provided by Southwest Research Institute (SwRI) were included in the guidelines and recommended best practice. Use of PIRT is a flexible approach to outline and describe model hierarchy and parameters, assess their relative importance, and communicate or illustrate this to the project team or reviewers and is now included as a recommended tool for use.

  • Development of a ‘Dashboard’ chart: A simple, one-page chart in presentation format was developed, suitable for succinctly summarizing the status of ICME V&V planning and execution for a specific project. The ‘Dashboard’ reflects the status of checklist items, TML, and any risk mitigation items in a format suitable for reporting and presentation.

  • Application examples were expanded and improved: A more detailed example of use of the ‘Risk vs. Consequences’ approach for prediction of forging residual stress predictions was included. A detailed example of uncertainty assessment for an ICME computational model for aluminum was expanded from the prior version and related to the TML assessment guide.

  • Alignment of the guidelines and recommended best practice with current gated processes: Background and supporting text was revised, and figures showing alignment of the recommended best practice with TRL, MRL, IPD, and the Department of Defense (DoD) Acquisition Guide were revised and improved.

One key conclusion of the effort to revise and update these guidelines was that such guidelines are, in fact, essential to the successful communication of V&V activities across the supply chain and between engineering, manufacturing, and product disciplines. The perception of what constitutes adequate V&V for ICME applications was observed to vary significantly amongst practitioners and was often based on whether the practitioner was a supplier, an original equipment manufacturer, or a customer. The adoption of a common approach to V&V was found to facilitate the successful development and implementation of ICME methodologies.

Availability of supporting data

The data sets supporting the results of this article are included within the article and its additional files.