Advertisement

Credibly reaching a reliability target using a model initially constructed by expert elicitation

  • Lawrence E PadoEmail author
Research
  • 1k Downloads

Abstract

The Defense Advanced Research Projects Agency Defense Science Office (DARPA/DSO) is sponsoring Open Manufacturing (OM), an initiative to develop new technologies, new computational tools, and rapid qualification to accelerate the manufacturing innovation timeline. Certification Methodology to Transition Innovation (CMTI), an OM program, has developed a methodology to quantify the effect of manufacturing variability on product performance to address the risk to cost and performance associated with failure to take manufacturing capability and material and fabrication/assembly variation into account early in the design process. An important aspect of this program is the use of Bayesian networks (BN) to evaluate risk. The BN is used as a graphical representation of the contributing factors that lead to manufacturing defects. The reliability of the final product is then analyzed using the contributing factors. There are many types of programs where there is little relevant data to support the probabilities needed to populate the BN model. This is very likely the case for new programs or at the end of long programs when obsolescence challenges servicing a product when original vendors are no longer in business. In these cases, probabilities must be obtained from expert opinion using a technique called expert elicitation. Even under objective ‘Good Faith’ opinions, the expert himself has a lot of uncertainty in that opinion. This paper details an approach to obtaining credible model output based on the idea of having a hypothetical expert whose unconscious bias influences the model output and discovering and using countermeasures to find and prevent these biases. Countermeasures include replacing point probabilities with beta distributions to incorporate uncertainty, 95% confidence levels, and using a multitude of different types of sensitivity analyses to draw attention to potential trouble spots. Finally, this paper uses a new technique named ‘confidence level shifting’ to optimally reduce epistemic uncertainty in the model. Taken together, the set of tools described in this paper will allow an engineer to cost effectively determine which areas of the manufacturing process are most responsible for performance variance and to determine the most effective approach to reducing that variance in order to reach a target reliability.

Keywords

Credibility Expert elicitation Confidence level shifting Monte Carlo Uncertainty quantification Targeted testing Unitized testing Uncertainty reduction Epistemic uncertainty Reliability targets 

Abbreviations

Δ

delta, change in value

A

beta distribution parameter expressing the number of flawed examples

B

beta distribution parameter expressing the number of flawless examples

BN

Bayesian networks

CL

confidence level

CLS

confidence level shifting

CMTI

Certification Methodology to Transition Innovation

DARPA/DSO

Defense Advanced Research Projects Agency Defense Science Office

GSA

global sensitivity analysis

K

expert confidence in estimate in terms of equivalent prior sample size

Mode

the most likely probability of a flaw

NRT

negative result test

OM

Open Manufacturing

P

proportion of flaws in the beta distribution

Pi

probability of node i inducing or failing to detect a defect

POF

probability of failure

QA

quality assurance

RT

radial thickening

S95%CLi

sensitivity of the 95% CL of model output due to change in mode of node i

SDi

derivative-based sensitivity measure

Si

effect due to variable i

STi

total effect due to variable i

Xi

model parameter

Y

model output

Notes

Acknowledgements

This paper is sponsored by Defense Advanced Research Projects Agency, Defense Sciences Office under the Open Manufacturing Program, ARPA Order No. S587/00, Program Code 2D10, issued by DARPA/CMO under contract no. HR 0011-12-C-0034. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressly or implied, of the Defense Advanced Research Projects Agency of the U.S. Government. This paper was approved for public release, distribution unlimited as 14-00070-EOT.

Supplementary material

40192_2014_20_MOESM1_ESM.gif (23 kb)
Authors’ original file for figure 1
40192_2014_20_MOESM2_ESM.gif (9 kb)
Authors’ original file for figure 2
40192_2014_20_MOESM3_ESM.gif (39 kb)
Authors’ original file for figure 3
40192_2014_20_MOESM4_ESM.gif (39 kb)
Authors’ original file for figure 4
40192_2014_20_MOESM5_ESM.gif (26 kb)
Authors’ original file for figure 5
40192_2014_20_MOESM6_ESM.gif (11 kb)
Authors’ original file for figure 6
40192_2014_20_MOESM7_ESM.gif (17 kb)
Authors’ original file for figure 7
40192_2014_20_MOESM8_ESM.gif (12 kb)
Authors’ original file for figure 8
40192_2014_20_MOESM9_ESM.gif (54 kb)
Authors’ original file for figure 9
40192_2014_20_MOESM10_ESM.gif (23 kb)
Authors’ original file for figure 10
40192_2014_20_MOESM11_ESM.gif (29 kb)
Authors’ original file for figure 11
40192_2014_20_MOESM12_ESM.gif (31 kb)
Authors’ original file for figure 12
40192_2014_20_MOESM13_ESM.gif (25 kb)
Authors’ original file for figure 13
40192_2014_20_MOESM14_ESM.gif (26 kb)
Authors’ original file for figure 14
40192_2014_20_MOESM15_ESM.gif (17 kb)
Authors’ original file for figure 15
40192_2014_20_MOESM16_ESM.gif (16 kb)
Authors’ original file for figure 16
40192_2014_20_MOESM17_ESM.gif (16 kb)
Authors’ original file for figure 17
40192_2014_20_MOESM18_ESM.gif (27 kb)
Authors’ original file for figure 18

References

  1. 1.
    Renieri G: High performance stiffened panel design concept. Annual Report. Office of Naval Research (ONR), Arlington, VA; 2013.Google Scholar
  2. 2.
    Koller D, Friedman N: Probabilistic graphical models: principles and techniques. Press, MIT, Cambridge; 2009.Google Scholar
  3. 3.
    Hahn GL, Pado LE, Thomas MJ, Liguore SL: Application of risk quantification approach to aerospace manufacturing using Bayesian networks. Paper presented at AIAA SciTech 2014. National Harbor, Maryland; 2014.Google Scholar
  4. 4.
    Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, Saisana M, Tarantola S: Global sensitivity analysis. The Primer, Wiley, John & Sons, , Incorporated; 2008.Google Scholar
  5. 5.
    Kruschke J: Doing Bayesian data analysis. Press, Academic, New York; 2011.Google Scholar
  6. 6.
    Kruschke J: Beta distribution parameterized by mode instead of mean. 2012.Google Scholar
  7. 7.
    Haldar A, Mahadevan S: Probability, reliability, and statistical methods in engineering design. Wiley, John & Sons, Incorporated, New York; 2000.Google Scholar
  8. 8.
    Milton JS, Arnold JC: Probability and statistics in the engineering and computing sciences. Elsevier Academic Press, Burlington, MA; 1986.Google Scholar
  9. 9.
    Abrahamson N: Seismic hazard assessment: problems with current practice and future developments Proceedings. First European Conference on Earthquake Engineering and Seismology, Geneva, Switzerland; 2006.Google Scholar

Copyright information

© Pado.; licensee Springer. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.The Boeing CompanyBerkeleyUSA

Personalised recommendations