Update to recommended best practice for verification and validation of ICME methods and models for aerospace applications
Rigorous verification and validation practices are regarded essential for successful implementation of Integrated Computational Materials Engineering (ICME) for aerospace applications. Guidelines and a recommended best practice for ICME verification and validation were developed in 2011 under Air Force Research Laboratory sponsorship and were published in 2012. These included general guidelines for ICME verification and validation, a recommended best practice, practitioner aides in the form of checklists for planning and status assessment, a Tool Maturity Level assessment guide, and a recommended approach for risk vs. consequence assessment for specific ICME applications. ICME development programs and expert practitioners have since reviewed and used the guidelines and provided constructive feedback for improvements.
This feedback was used to generate a comprehensive revision to the original guidelines and recommended best practice, as well as to the corresponding practitioner aides. The System-Level and Model-Level Checklists were simplified and improved, as was the Tool Maturity Level assessment guide. A template for a one-page summary chart was also generated to facilitate communication of status.
Experience with several ICME-related programs and expert practitioners reinforced the conclusion that guidelines and a recommended best practice are important tools to ensure that rigorous, comprehensive verification and validation practices are employed during ICME development and implementation efforts.
KeywordsICME Verification Validation Tool maturity level System-level checklist Model-level checklist
Air Force Research Laboratory
American Society of Mechanical Engineers
Defense Advanced Research Projects Agency
Department of Defense
Integrated Computational Materials Engineering
Integrated Product Development (sometimes …Deployment)
Metals Affordability Initiative
Manufacturing Readiness Level
Phenomena Identification and Ranking Technique
Southwest Research Institute
Tool Maturity Level
Technology Readiness Level
verification and validation
Background and hypothesis
Utilize current, recognized practices for V&V as the basis. This was predominately the American Society of Mechanical Engineers (ASME) V&V Guide - 2006 .
Facilitate alignment with established, gated practices for product and technology development. These included the Technology and Manufacturing Readiness Level processes (TRL and MRL, respectively) and both the Integrated Product Development (IPD) and Defense Acquisition Guidebook processes for system or product development.
Provide simple, useful guidance and tools to aid practitioners in planning and assessing V&V activities and results for ICME development.
The rationale and approach for developing these guidelines and practitioner aides were described in a journal article in 2012 . The initial versions of the recommended best practice, practitioner aides such as V&V checklists, a Tool Maturity Level (TML) assessment approach, and a recommended approach to assess risk vs. consequences of ICME application were appended as additional files accessible through the IMMI journal reference.
It was expected that the initial guidelines and practitioner aides would benefit from a revision cycle following some application experience. Several AFRL- and Defense Advanced Research Projects Agency (DARPA)-sponsored programs have since reviewed and/or applied the guidelines. Feedback from these programs and from individual experts was used to generate a comprehensive revision to the original guidelines and practitioner aides. This effort was performed under AFRL sponsorship of the Metals Affordability Initiative (MAI) program ‘GE-12: Verification and Validation (V&V) of ICME Models and Methods’ (Agreement Number: FA8650-08-2-5251).
Methods and approach
Feedback was solicited from numerous individuals engaged in ICME development programs representing aerospace propulsion companies and from within AFRL. Participants represented three programs where the guidelines and practitioner aides were used directly, in whole or in part. In these cases, technology and/or intended product applications were well defined, as was the intended role of the ICME elements under development. Responses were also solicited from selected individuals with prior ICME development and implementation experience. Feedback was collected over an 18-month period, from May 2012 through October 2013.
Feedback was generally consistent and focused on the need to simplify wording, reduce the number of line items in the practitioner aides, and clarify the recommended process flow for V&V planning and execution. The feedback was collectively reviewed, consolidated, and used to generate revisions to the ICME V&V guidelines, the recommended best practice, and the associated practitioner aides. The text document containing the revised guidelines and recommended best practice  and the associated Excel™ file containing the revised practitioner aides  were released by AFRL for public distribution in January 2014. These references are appended as Additional files 1 and 2.
The flow chart indicates the key steps in the process and where the practitioner aides may be helpful. The flow chart also indicates the expected iterative nature of developing an ICME V&V plan based on a specific technology or product application. A more detailed flow chart, and associated descriptions of the steps involved, is presented in the appended files [3,4].
System-Level and Model-Level Checklists: These practitioner aides were simplified in wording and number of line items. The supporting text was significantly revised and expanded to more clearly communicate intent and recommended process for use of this checklist.
TML assessment guide: This practitioner aide was revised by adding numerous supporting comments in the Excel™ comment tabs, adjusting some of the category headers based on feedback, and by adding some line items in the category boxes for selected TML levels.
Recommended Best Practice for ICME V&V: The two recommended process flow charts were revised. The simplified (‘3-step’) process flow chart was updated to illustrate the iterative and interactive nature recommended for key steps in the V&V planning process. The detailed process flow chart was revised to improve flow and reflect the expected actual sequence of activities in a V&V planning process. In addition, the step-by-step instructions were updated and recommended activities clarified.
Use of Phenomena Identification and Ranking Technique (PIRT) (note that often ‘Table’ is used instead of ‘Technique’ in this acronym): Instructions and an ICME-relevant example for the use of this technique provided by Southwest Research Institute (SwRI) were included in the guidelines and recommended best practice. Use of PIRT is a flexible approach to outline and describe model hierarchy and parameters, assess their relative importance, and communicate or illustrate this to the project team or reviewers and is now included as a recommended tool for use.
Development of a ‘Dashboard’ chart: A simple, one-page chart in presentation format was developed, suitable for succinctly summarizing the status of ICME V&V planning and execution for a specific project. The ‘Dashboard’ reflects the status of checklist items, TML, and any risk mitigation items in a format suitable for reporting and presentation.
Application examples were expanded and improved: A more detailed example of use of the ‘Risk vs. Consequences’ approach for prediction of forging residual stress predictions was included. A detailed example of uncertainty assessment for an ICME computational model for aluminum was expanded from the prior version and related to the TML assessment guide.
Alignment of the guidelines and recommended best practice with current gated processes: Background and supporting text was revised, and figures showing alignment of the recommended best practice with TRL, MRL, IPD, and the Department of Defense (DoD) Acquisition Guide were revised and improved.
One key conclusion of the effort to revise and update these guidelines was that such guidelines are, in fact, essential to the successful communication of V&V activities across the supply chain and between engineering, manufacturing, and product disciplines. The perception of what constitutes adequate V&V for ICME applications was observed to vary significantly amongst practitioners and was often based on whether the practitioner was a supplier, an original equipment manufacturer, or a customer. The adoption of a common approach to V&V was found to facilitate the successful development and implementation of ICME methodologies.
Availability of supporting data
The data sets supporting the results of this article are included within the article and its additional files.
The authors would like to thank John Warren (GE) and David Riha (SwRI) for their contributions to the development of this material.
- 1.The American Society of Mechanical Engineers (2006) Guide for verification and validation in computational solid mechanics. ASME V&V 10-2006. ASME International, New YorkGoogle Scholar
- 2.Cowles BA, Backman DG, Dutton RE (2012) Verification and validation of ICME methods and models for aerospace applications. Integrating Materials and Manufacturing Innovation 1:2. http://www.immijournal.com/content/1/1/2
- 3.Cowles BA, Backman DG (2013) Verification and validation of ICME methods and models for aerospace applications – V&V guidelines and recommended best practice, revision 1. Summary document prepared for AFRL/RXM. Cowles Consulting, LLC, Tolland. Appended as Additional file 1Google Scholar
- 4.Cowles BA, Backman DG (2013) Verification and validation of ICME methods and models for aerospace applications – V&V guidelines and recommended best practice, revision 1. Spreadsheet tools and practitioner aides prepared for AFRL/RXM. Cowles Consulting, LLC, Tolland. Appended as Additional file 2Google Scholar
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.