Skip to main content

A case for trading risk in complex conceptual design trade studies

Abstract

Complex conceptual system design trade studies traditionally consider risk after a conceptual design has been created. Further, one person is often tasked with collecting risk information and managing it from each subsystem. This paper proposes a method to explicitly consider and trade risk on the same level as other important system-level variables during the creation of conceptual designs in trade studies. The proposed risk trading method advocates putting each subsystem engineer in control of risk for each subsystem. A risk vector is proposed that organizes many different risk metrics for communication between subsystems. A method of coupling risk models to dynamic subsystem models is presented. Several risk visualization techniques are discussed. A trade study example is presented based upon a simplified spacecraft model. Results from introducing the risk trading methodology into a simulated Collaborative Design Center are presented. The risk trading method offers an approach to more thoroughly consider risk during the creation of conceptual designs in trade studies.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

References

  • Andersson P (1996) A semi-analytic approach to robust design in the conceptual design phase. Res Eng Design 8:229–239

    Article  Google Scholar 

  • Andersson P (1997) On robust design in the conceptual design phase: a qualitative approach. J Eng Design 8(1):75–90

    Article  Google Scholar 

  • Benjamin JL, Pate-Cornell ME (2004) Risk chair for concurrent design engineering: satellite swarm illustration. J Spacecr Rockets 41(1):51–59

    Article  Google Scholar 

  • Bennett R, Roberts B (2000) Risk management for the nasa/jpl genesis mission: a case study. In: Proceedings of the 2000 international council on systems engineering conference, INCOSE

  • Bonano EJ, Hora SC, Keeney RL, von Winterfeldt D (1990) Elicitation and use of expert judgment in performance assessment for high-level radioactive waste repositories. Tech. Rep. NUREG/CR-5411, Nuclear Regulatory Commission, Washington

  • Browning TR, Eppinger SD (2000) Modeling the impact of process architecture on cost and schedule risk in product development. Sloan Manag Rev WPN 4050

  • Browning TR, Eppinger SD (2002) Modeling impacts of process architecture on cost and schedule risk in product development. IEEE Trans Eng Manag 49(4):428–442

    Article  Google Scholar 

  • Browning TR, Deyst JJ, Eppinger SD, Whitney DE (2002) Adding value in product development by creating information and reducing risk. IEEE Trans Eng Manag 49:443–458

    Article  Google Scholar 

  • Charania AC, ohn E Bradford J, Olds JR, Graham M (2002) System level uncertainty assessment for collaborative rlv design. In: Second modeling and simulation subcommittee joint meeting

  • Clarkson P, Simons C, Eckert C (2004) Predicting change propagation in complex design. J Mech Des 126:788

    Article  Google Scholar 

  • Clemen RT, Winkler RL (1999) Combining probability distributions from experts in risk analysis. Risk Anal 19(2):187–204

    Google Scholar 

  • Cooke RM (1991) Experts in uncertainty: opinions and subjective probability in science. Oxford University Press, New York

    Google Scholar 

  • Cornford SL, Dunphy J, Feather MS (2002) Optimizing the design of spacecraft systems using risk as currency. In: IEEE aerospace conference, ddptool.jpl.nasa.gov

  • Cornford SL, Feather MS, Jenkins JS (2006) Intertwining risk insights and design decisions. In: Eigth international conference on probabilistic safety assessment and management

  • Department of Defense (1980) Procedures for performing failure mode, effects, and criticality analysis. MIL-STD-1629A.

  • Deutsch MJ, Nichols JS (2000) Advanced approach to concept and design studies for space missions. Astrophys Space Sci 273:201–206

    Article  Google Scholar 

  • Dezfuli H, Youngblood R, Reinert J (2007) Managing risk within a decision analysis framework. In: Second international association for the advancement of space safety conference, IAASS

  • Du X, Chen W (2000) Towards a better understanding of modeling feasibility robustness in engineering design. ASME J Mech Des 122(4):385–394

    Article  Google Scholar 

  • FAA (2006) National airspace system engineering manual, 3rd edn. Federal Aviation Administration ATO Operations Planning

  • Ford RB, Barkan P (1995) Beyond parameter design—a methodology addressing product robustness at the concept formation stage. In: Proceedings of the national design engineering conference

  • Grantham-Lough K, Stone R, Tumer IY (2007) The risk in early design method. J Eng Des 20:155–173

    Article  Google Scholar 

  • Guikema SD, Pate-Cornell ME (2004) Bayesian analysis of launch vehicle success rates. J Spacecr Rockets 41(1):93–102

    Article  Google Scholar 

  • Hardman DK, Ayton P (1997) Arguments for qualitative risk assessment: the star risk adviser. Expert Syst 14:24–36

    Article  Google Scholar 

  • Hora SC (1992) Acquisition of expert judgment: Examples from risk assessment. J Energy Eng 118:136–148

    Article  Google Scholar 

  • IEEE (1990) IEEE standard computer dictionary: a compilation of IEEE standard computer glossaries. IEEE, New York

  • International Electrotechnical Commission (1990) International standard IEC 61025 fault tree analysis

  • International Organization for Standardization (1997) ISO 10628: Flow diagrams for process plants—general rules

  • International organization for standardization (2009) ISO 31000:2009 risk management—principles and guidelines

  • Jensen D, Tumer IY, Kurtoglu T (2009) Flow state logic (fsl) for analysis of failure propagation in early design. In: Proceedings of the ASME design engineering technical conferences, international design theory and methodology conference, IDETC/CIE2009, San Diego

  • Ji H, Yang MC, Honda T (2007) A probabilistic approach for extracting design preferences from design team discussion. In: Proceedings of the ASME 2007 international design engineering technology conferences and computers in information and engineering conference (IDETC/CIE2007), IDETC/CIE, Las Vegas, NV

  • Keeney RL, von Winterfeldt D (1989) On the uses of expert judgment on complex technical problems. IEEE Trans Eng Manag 36(2):219–229

    Article  Google Scholar 

  • Keeney RL, von Winterfeldt D (1991) Eliciting probabilities from experts in complex technical problems. IEEE Trans Eng Manag 38:191–201

    Article  Google Scholar 

  • Krus D, Grantham-Lough K (2007) Applying function-based failure propagation in conceptual design. In: The Proceedings of the ASME design engineering technical conferences, international design theory and methodology conference, Las Vegas, NV

  • Kurtoglu T, Tumer IY (2008) A graph-based fault identification and propagation framework for functional design of complex systems. J Mech Des 30(5)

  • Kurtoglu T, Tumer IY, Jensen D (2010) A function failure reasoning methodology for evaluation of conceptual system architectures. Res Eng Des 21(4):209

    Article  Google Scholar 

  • Lough KG, Stone RB, Tumer IY (2008) Implementation procedures for the risk in early design (red) method. J Ind Syst Eng 2(2):126–143

    Google Scholar 

  • Lough KG, Van Wie M, Stone R, Tumer I (2009) Promoting risk communication in early design through linguistic analyses. Res Eng Des 20(1):29–40

    Article  Google Scholar 

  • Martin JD, Simpson TW (2006) A methodology to manage system-level uncertainty during conceptual design. ASME J Mech Des 128:959–968

    Article  Google Scholar 

  • McCormick NJ (1981) Reliability and risk analysis (methods and nuclear power applications). Academic Press, London

    Google Scholar 

  • McManus HL, Warmkessel JM (2004) Creating advanced architectures for space systems: emergent lessons from new processes. J Spacecr Rockets 41:69–75

    Article  Google Scholar 

  • McManus HL, Hastings DE, Warmkessel JM (2004) New methods for rapid architecture selection and conceptual design. J Spacecr Rockets 41(1):10–19

    Article  Google Scholar 

  • Mehr AF, Tumer IY (2006) Risk-based decision-making for managing resources during the design of complex space exploration systems. J Mech Des 128:1014–1022

    Article  Google Scholar 

  • Merkhofer MW (1987) Quantifying judgmental uncertainty: methodology, experience, and insights. IEEE Trans Syst Man Cybern 17:741–752

    Google Scholar 

  • Meshkat L (2007) A holistic approach for risk management during design. In: IEEE aerospace conference

  • Meshkat L, Weiss KA, Luna M, Leveson N (2006) Supporting concurrent engineering in JPL’s advanced project design team using a systems engineering development environment. In: In the proceedings of virtual concept

  • Mosleh A, Beier VM, Apostolakis G (1987) A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliab Eng Syst Saf 20:63–85

    Article  Google Scholar 

  • NASA (1995) NASA systems engineering handbook. NASA

  • Oberto RE, Nilsen E, Cohen R, Wheeler R, DeFlorio P, Borden C (2005) The NASA exploration design team: blueprint for a new design paradigm. In: Proceedings of the 2005 Aerospace Conference, IEEE, no. 8957662 in IEEE Conferences, pp 4398–4405

  • Osburg J, Mavris D (2005) A collaborative design environment to support multidisciplinary conceptual systems design. SAE Trans 114:1508–1516

    Google Scholar 

  • Otway H, von Winterfeldt D (1992) Expert judgment in risk analysis and management: process, context, and pitfalls. Risk Anal 12:83–93

    Article  Google Scholar 

  • Papalambros PY, Wilde DJ (2000) Principles of optimal design: modeling and computation. Cambridge University Press, Cambridge

    MATH  Book  Google Scholar 

  • Parkin KL, Sercel JC, Liu MJ, Thunnissen DP (2003) Icemaker: an excel-based environment for collaborative design. In: In the Proceedings of IEEE Aerospace Conference

  • Phoenix Integration Inc (2008) PHX Model Center. http://www.phoenix-int.com/software/phx_modelcenter.php

  • Reich Y, Ziv Av A (2005) Robust product concept generation. In: International conference on engineering design ICED05

  • Ross AM, Hastings DE, Warmkessel JM, Diller NP (2004) Multi-attribute tradespace exploration as front end for effective space system design. J Spacecr Rockets 41(1):20–29

    Article  Google Scholar 

  • Russell JS, Skibniewski MJ (1988) Decision criteria in contractor prequalification. J Manag Eng 4(2):148–164

    Article  Google Scholar 

  • Shishko R (2000) The proliferation of pdc-type environments in industry and universities. In: Proceedings of the 2nd European systems engineering conference, EuSEC

  • Stamanis DH (2003) Failure modes and effects analysis: FMEA from theory to execution, 2nd edn. ASQ Quality Press, Milwaukee

  • Stamatelatos M, Dezfuli H, Apostolakis G (2006) A proposed risk-informed decision-making framework for nasa. In: 8th international conference on probabilistic safety assessment and management

  • Stone RB, Tumer IY, Wie MV (2005) The function-failure design method. J Mech Des 127(3):397–407

    Article  Google Scholar 

  • Stone RB, Tumer IY, Stock ME (2006) Linking product functionality to historical failures to improve failure analysis in design. Res Eng Des 16(2):96–108

    Google Scholar 

  • Stump GM, Lego S, Yukish M, Simpson TW, Donndelinger JA (2009) Visual steering commands for trade space exploration: user-guided sampling with example. J Comp Inform Sci Eng 9(4):044,501:1–10

    Google Scholar 

  • Taguchi G (1986) Introduction to quality engineering. Quality Resources, White Plains

    Google Scholar 

  • Taguchi G (1993) Taguchi on Robust Technology Development. ASME, New York

    Book  Google Scholar 

  • Thunnissen DP (2003) Uncertainty classification for the design and development of complex systems. In: 3rd annual predictive methods conference

  • Thunnissen DP (2004) Balancing cost, risk, and performance under uncertainty in preliminary mission design. In: AIAA space conference

  • Thunnissen DP, Tsuyuki GT (2004) Margin determination in the design and development of a thermal control system. In: 34th international conference on environmental systems (ICES)

  • Tumer IY, Stone RB (2003) Mapping function to failure mode during component development. Res Eng Des 4(1):25–33

    Google Scholar 

  • Ullman DG (2001) Robust decision-making for engineering design. J Eng Des 12(1):3–13

    MathSciNet  Article  Google Scholar 

  • Ullman DG (2003) The mechanical design process, 3rd edn. McGraw-Hill, New York

    Google Scholar 

  • Van Bossuyt DL, Tumer IY (2010) Toward understanding collaborative design center trade study software upgrade and migration risks. In: Proceedings of the ASME 2010 international mechanical engineering congress and exposition IMECE2010, ASME, Vancouver

  • Van Bossuyt DL, Wall S, Tumer I (2010) Towards risk as a tradeable parameters in complex systems design trades. In: Proceedings of the ASME 2010 International design engineering technology conferences and computers in information and engineering conference (IDETC/CIE2010), ASME, Montreal, pp DETC2010–29,016

  • Villemeur A (2000) Reliability, availability, maintainability, and safety assessment. Willey, New Jersey

    Google Scholar 

  • Wertz JR, Larson WJ (1999) Space mission analysis and design. Springer, Berlin

    Google Scholar 

  • Ziv Av A, Reich Y (2005) Sos-subjective objective system for generating optimal product concepts. Des Stud 26(5):509–533

    Article  Google Scholar 

Download references

Acknowledgments

This research was carried out in part at JPL, Caltech, under contract with NASA. Special thanks goes to Scott Ragon, Taurik Elgabrowny, and others at Phoenix Integration Inc. for donating software and providing technical support, and Steve Cornford at JPL for providing valuable feedback and inspiration. The study protocol was reviewed and approved by the Institutional Review Board, Study 4611, at Oregon State University. The opinions and findings of this work are the responsibility of the authors and do not necessarily reflect the views of the sponsors or collaborators.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Douglas L. Van Bossuyt.

Appendices

Appendix 1: Subsystem development

To represent the spacecraft, four representative subsystems including Communication, Data Handling, Attitude Control, and Power were chosen. The Communication Subsystem is a function-based model that accepts user input for the Antenna Size and Frequency Downlink variables. Function-based subsystem models are function-driven over a range of numeric inputs, while component-based subsystems have a predefined, limited selection of potential subsystem components. Antenna size can range from 1 to 4, and Frequency Downlink can range from 1 to 18, including decimal values. Both of the user input fields have corresponding instructions for the user to maintain input values between the allowable ranges. The Communication Subsystem Power requirements, Mass, and Cost output variables were computed using the formulas shown in Eqs. 4, 5, and 6, respectively.

$$ \hbox{Power}= -\hbox{Antenna Size} + 0.6 \times\hbox{Frequency Downlink} + 3 $$
(4)
$$ \hbox{Mass} = \hbox{Antenna Size} \times 2.5 + 2 $$
(5)
$$ \hbox{Cost} = \hbox{Antenna Size} \times 0.75 + \hbox{Frequency Downlink} \times 0.1 $$
(6)

The Data Handling Subsystem is a component-based model that contains two user inputs in the form of drop-down selection boxes. The first user input, System Complexity, has the options of “simple,” “typical,” and “complex.” The other user input is Spacecraft Bus Configuration which allows the user to select either “one unit,” “two unit,” or “integrated” which refer to the spacecraft having one or two primary computing units and distributed subsystem computers, or an integrated unit that handles all command and data handling functionality. The resulting Data Handling subsystem outputs are shown in Table 2.

Table 2 Data handling subsystem input and output variables

The Attitude Control Subsystem is a component-based model that gives the user control over two inputs via drop-down selection boxes. The inputs are “Stability Method” and “Pointing Method.” Table 3 displays the full range of user-selectable components and the corresponding output variable values.

Table 3 Attitude control subsystem input and output variables

The Power Subsystem is driven by a component-based model that has two inputs, namely “Power Source” and “Energy Source,” which are controllable via drop-down selection boxes. Table 4 presents the range of possible user-selectable input variable combinations and their corresponding output variables. Unlike the other three subsystems, the Power output variable for the Power Subsystem indicates how much power is available to the entire spacecraft system from the power produced within the Power Subsystem.

Table 4 Power subsystem input and output variables

In addition to the four participant-controlled subsystems, a Payload Subsystem was also developed from Wertz and Larson (Wertz and Larson 1999). It is used only to set the mission objectives and requirements. The two possible payloads consist of a weather and navigation package. Only one payload package is selectable at any given time. The Payload Subsystem outputs power, mass, and cost variables. It also produces data on system constraints due to the payload. Table 5 presents the two payload choices and corresponding output data.

Table 5 Payload subsystem input and output variables

Appendix 2: Problem statements

The riskless trade study session used a simple navigation satellite problem. The problem statement is as follows:

This satellite is designed as a navigation satellite to add to the GPS network allowing GPS units to acquire more accurate data on Earth. It carries equipment on board to support its mission. Because of this, the following constraints are given for the mission:

  • POWER SUBSYSTEM Power Source: photovoltaic

  • COMMUNICATIONS SUBSYSTEM: Frequency downlink: 18

  • DATA HANDLING SUBSYSTEM: Required processing: 110

  • TOTAL SPACECRAFT: Maximum mass: 30 Maximum cost: 18

The trade study session conducted using the risk trading methodology used a simple weather satellite problem. The problem statement is as follows:

This satellite is designed as a weather satellite to monitor the climate on Earth and carries equipment on board to support its mission. Because of this, the following constraints are given for this mission:

  • POWER SUBSYSTEM Energy Storage: primary and secondary battery

  • DATA HANDLING SUBSYSTEM: Spacecraft bus: 2 units Required processing: 105

  • TOTAL SPACECRAFT: Maximum mass: 27 Maximum cost: 17

Appendix 3: Questionnaire questions

Following each trade study session, participants were asked to fill out a questionnaire individually. The following questions were common to both trade studies.

  • Rank the ease of use of each subsystem model on an Easy (1) to Hard (5) scale:

    • Attitude control

    • Data handling

    • Power

    • Communications

  • Indicate the ease of use of the two types of subsystem models on an Easy (1) to Hard (5) scale:

    • Component-based

    • Function-based

Additional questionnaire questions were tailored to the risk trading session including:

  • Describe any difficulties you encountered while understanding and using the subsystem risk models

  • How did you find the transition from conducting trade studies without risk models to trade studies with risk models on an Easy (1) to Hard (5) scale?

  • Indicate which set of models produced results in which you feel more confident on a Confident in no-risk model results (1) to confident in models with risk results (5) scale

  • Indicate the ease of understanding risk data for each risk visualization technique on an Easy (1) to Hard (5) scale:

    • Fever charts

    • Glyph plots

    • Parallel axis

    • Numeric data

    • Dynamic fault tree

  • Is there anything that should have been done differently when transitioning from trade study models not containing risk information to trade study models with components?

  • Do you have any additional comments about the study or anything else you wish to convey to the researchers?

Appendix 4: Group discussion questions

Group discussion followed completion of the System Design Report and the questionnaire in both trade study sessions. The following questions were repeated at the end of both sessions:

  • Were any of the subsystem models hard to understand and use? Were any particularly easy?

  • Did you prefer component-based or function-based subsystem models?

The following questions were used in the group discussion only for the second trade study:

  • Did you encounter any difficulties using subsystem models with risk data?

  • Were you able to understand the graphical representations of risk? Which did you prefer? (Glyph plot, fever chart, parallel axis plot, dynamic fault tree)

  • Is there anything that should have been done differently when transitioning from trade study models not containing risk information to trade study models with risk components?

  • Do you have any additional comments about the study or anything else you wish to convey to the researchers?

Appendix 5: Work product template

At the end of both trade study sessions, participants completed brief reports about the work that they had just completed. The following free entry form was provided to the participants:

  • Subsystem

  • Design Decisions

  • Rationale

  • Comments

Most participants wrote a paragraph or more for each of the last three questions.

Appendix 6: Questionnaire results

Relevant questionnaire responses are aggregated in this appendix. Identifying information has been removed, and data have been anonymized.

Describe any difficulties you encountered while understanding and using subsystem risk models

  • The risk models were extremely helpful and intuitive.

  • The risk models were easy to understand but mitigating design problems was difficult.

  • The only challenge was to observe how design changes propagated through the subsystem and system models.

How did you find the transition from conducting trade studies without risk models to trade studies with risk models

  • Risk is just one more thing to analyze. Engineers should already be doing this.

  • Trading risk was straight forward.

  • The risk trading method provided more perspective and helps me to feel confident in the final design.

  • Risk adds another variable for consideration that can make it more difficult to find a satisfactory solution.

  • The risk method is more all-encompassing.

  • Risk adds another parameter and is not hard to deal with.

Indicate which set of models produced results in which you feel more confident

  • Knowing that design decisions are backed by the science of risk methods such as (FMEA) makes me very confident in our design choices.

Is there anything that should have been done differently when transitioning from trade study models not containing risk information to trade study models with risk components?

  • No.

  • The brief training was straightforward.

  • The transition was straightforward.

  • A better understanding of the trade-offs between risk metrics and other system variables would be useful.

Do you have any additional comments about the study or anything else you wish to convey to the researchers?

  • The risk trading method and dynamic (FMEA) model are big improvements over existing methods. The method provides for another layer of reliability in the design.

Appendix 7: Group discussion results

Relevant group discussion responses are aggregated in this appendix. Identifying information has been removed ,and data have been anonymized.

  • Using the risk trading method was not harder than not using the method.

  • I liked the risk trading method. It validates that there is more to the model.

  • The resulting design is more complete when using the risk trading method. The resulting design is safer.

  • The risk trading method was as easy to use as standard trade study methods. It was more complex but not more difficult.

  • I would be more comfortable to show my boss the conceptual design created using the risk trading method. (three participants stated this)

  • Using the risk trading method helped me to make design decisions more comfortably.

  • It makes sense from an engineering perspective that there is a trade-off between traditional variables such as power, mass, and cost, and engineering risk metrics.

  • I am more confident in conceptual designs created using the risk trading method.

  • I prefer using the risk trading method over not using the method.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Van Bossuyt, D.L., Tumer, I.Y. & Wall, S.D. A case for trading risk in complex conceptual design trade studies. Res Eng Design 24, 259–275 (2013). https://doi.org/10.1007/s00163-012-0142-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00163-012-0142-0

Keywords

  • Trade study
  • Complex system design
  • Risk
  • Collaborative Design Center risk trading