Skip to main content
SpringerLink
Log in

A rule-based method for scalable and traceable evaluation of system architectures

  • Original Paper
  • Published:
Research in Engineering Design Aims and scope Submit manuscript

Abstract

Despite the development of a variety of decision-aid tools for assessing the value of a conceptual design, humans continue to play a dominant role in this process. Researchers have identified two major challenges to automation, namely the subjectivity of value and the existence of multiple and conflicting customer needs. A third challenge is however arising as the amount of data (e.g., expert judgment, requirements, and engineering models) required to assess value increases. This brings two challenges. First, it becomes harder to modify existing knowledge or add new knowledge into the knowledge base. Second, it becomes harder to trace the results provided by the tool back to the design variables and model parameters. Current tools lack the scalability and traceability required to tackle these knowledge-intensive design evaluation problems. This work proposes a traceable and scalable rule-based architecture evaluation tool called VASSAR that is especially tailored to tackle knowledge-intensive problems that can be formulated as configuration design problems, which is demonstrated using the conceptual design task for a laptop. The methodology has three main steps. First, facts containing the capabilities and performance of different architectures are computed using rules containing physical and logical models. Second, capabilities are compared with requirements to assess satisfaction of each requirement. Third, requirement satisfaction is aggregated to yield a manageable number of metrics. An explanation facility keeps track of the value chain all along this process. This paper describes the methodology in detail and discusses in particular different implementations of preference functions as logical rules. A full-scale example around the design of Earth observing satellites is presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. An earlier version of the VASSAR methodology was presented at the 2013 IEEE Aerospace Conference (Selva and Crawley, 2013), although that paper focused on the implications for space system architecting.

References

  • Antonsson EK, Cagan J (2001) Formal engineering design synthesis. Cambridge University Press, Cambridge. doi:10.1017/CBO9780511529627

    Book  Google Scholar 

  • Apgar H (2011) Cost estimating. In: Wertz JR, Everett DF, Puschell JJ (eds) Space mission engineering: the new SMAD. Microcosm, Hawthorne

  • Armacost RL, Componation PJ, Mullens MA, Swart WS (1994) An AHP framework for prioritizing customer requirements in QFD: an industrialized housing application. IIE Trans 26(4):72–79

    Article  Google Scholar 

  • Avigad G, Moshaiov A (2009) Interactive evolutionary multiobjective search and optimization of set-based concepts. IEEE Trans Syst Man Cybern Part B Cybern 39(4):1013–1027. doi:10.1109/TSMCB.2008.2011565

    Article  Google Scholar 

  • Bellman RE, Zadeh LA (1970) Decision-making in a fuzzy environment. Manag Sci 17(4):B-141

    Article  MathSciNet  Google Scholar 

  • Bonczek RH, Holsapple CW, Whinston AB (1981) A generalized decision support system using predicate calculus and network data base management. Oper Res 29(2):263–281. doi:10.2307/170020

    Article  MathSciNet  Google Scholar 

  • Buchanan BG, Shortliffe EH (1984) Rule-based expert systems: the MYCIN experiments of the Stanford Heuristic Programming Project. Language. Addison-Wesley, Reading

    Google Scholar 

  • Cameron B (2008) Value flow mapping: using networks to inform stakeholder analysis. Acta Astronaut 62(4–5):324–333. doi:10.1016/j.actaastro.2007.10.001

    Article  Google Scholar 

  • Campbell M, Cagan J, Kotovsky K (1999) A-design: an agent-based approach to conceptual design in a dynamic environment. Res Eng Design 11:172–192

  • Carlson-Skalak S, White M, Teng Y (1998) Using an evolutionary algorithm for catalog design. Res Eng Design 10:63–83

  • Chalmers University of Technology (2004) Use of P-band SAR for forest biomass and soil moisture retrieval. Retrieved from http://esamultimedia.esa.int/docs/gsp/completed/C16115ExS.pdf

  • Chandrasekaran B (1989) A framework for design problem-solving. Res Eng Design 1:75–86

    Article  Google Scholar 

  • Charnes A, Cooper WW (1957) Management models and industrial applications of linear programming. Manage Sci 4(1):38–91

    Article  MathSciNet  MATH  Google Scholar 

  • Clancey WJ (1987) Knowledge-based tutoring: the GUIDON program. In: MIT Press series in artificial intelligence. The MIT Press, Cambridge

  • Corkill DD (2003) Blackboard and multi-agent system & the future. In: Proceedings of the international lisp conference, New York, NY. Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Blackboard+and+Multi-Agent+Systems+&+the+Future#5

  • Crawley E, De Weck O, Eppinger S, Magee C, Moses J, Seering W et al (2004) The influence of architecture in engineering systems. In: Engineering systems monograph. Architecture

  • Das I, Dennis JE (1998) Normal-boundary intersection: a new method for generating the pareto surface in nonlinear multicriteria optimization problems. SIAM J Optim 8(3). doi:10.1137/S1052623496307510

  • De Condorcet M (1785) Essai sur l’application de l’analyse à la probabilité des décisions rendues à la probabilité des voix. De l’Imprimerie Royale, Paris

  • De Weck OL, Kim IY (2004) Adaptive weighted sum method for bi-objective optimization. In: Proceedings of the 45th AIAA/ASME/ASCE/, pp 1–13

  • Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6(2):182–197. doi:10.1109/4235.996017

    Article  Google Scholar 

  • Dincbas M (1980) A knowledge-based expert system for automatic analysis and synthesis in CAD. In: IFIP congress (pp 705–710)

  • Dobias AP (1990) Designing a mouse trap using the analytic hierarchy process and expert choice. Eur J Oper Res 48(1):57–65

    Article  Google Scholar 

  • Donaldson KM, Ishii K, Sheppard SD (2006) Customer value chain analysis. Res Eng Design 16(4):174–183. doi:10.1007/s00163-006-0012-8

    Article  Google Scholar 

  • Dori D (2002) Object-process methodology: a holistic paradigm. Springer, Berlin, pp 1–453

    Book  Google Scholar 

  • Duda RO, Gaschnig JG, Hart PE (1979) Model design in the PROSPECTOR consultant system for mineral exploration. In: Michie D (ed) Expert systems in the microelectronic age. Edinburgh University Press, Edinburgh, pp 153–167

    Google Scholar 

  • Durkin J (1990) Application of expert systems in the sciences. Ohio J Sci 90(5):171–179

    Google Scholar 

  • Engelmore R, Morgan T (eds) (1988) Blackboard systems. Addison-Wesley Publishing Company, Reading, pp 1–620

    Google Scholar 

  • Entekhabi D (2010) The soil moisture active passive (SMAP) mission. In: Proceedings of the IEEE, vol 98, pp 704–716. doi:10.1109/JPROC.2010.2043918

  • Erman L, Hayes-Roth F (1980) The Hearsay-II speech-understanding system: integrating knowledge to resolve uncertainty. ACM Comput Surv 12(2):213–253. http://dl.acm.org/citation.cfm?id=356816

  • Feigenbaum EA, Buchanan BG, Lederberg J (1971) On generality and problem solving: a case study using the DENDRAL program. In: Meltzer B, Michie D (eds) Machine Intelligence 6. Edinburgh, Scotland, pp 165–190

    Google Scholar 

  • Fenves S, Garrett J (1986) Knowledge based standards processing. Artif Intell Eng 1(1):3–14. doi:10.1016/0954-1810(86)90029-4

    Article  Google Scholar 

  • Fishburn PC (1974) Lexicographic orders, utilities and decision rules: a survey. Manag Sci 20(11):1442–1471

  • Fortin J, Dubois D, Fargier H (2008) Gradual numbers and their application to fuzzy interval analysis. IEEE Trans Fuzzy Syst 16(2):388–402. doi:10.1109/TFUZZ.2006.890680

    Article  Google Scholar 

  • Gasster S, Flaming GGM (1998) Overview of the conical microwave imager/sounder development for the NPOESS program. In: Geoscience and remote sensing …, vol 1, pp 268–270. IEEE. doi:10.1109/IGARSS.1998.702874

  • Geoffrion AM, Dyer JS, Feinberg A (1972) An interactive approach for multi-criterion optimization, with an application to the operation of an academic department. Manag Sci 19(4):357–368

  • Giarratano JC, Riley GD (2004) Expert systems: principles and programming, 4th edn. Course Technology. Retrieved from http://www.amazon.com/Expert-Systems-Principles-Programming-Fourth/dp/0534384471

  • Gologlu C, Mizrak C (2011) An integrated fuzzy logic approach to customer-oriented product design. J Eng Des 22(2):113–127. doi:10.1080/09544820903032519

    Article  Google Scholar 

  • Gutknecht O, Ferber J (2001) The MadKit agent platform architecture. In: Wagner T, Rana O (eds) Infrastructure for agents, multi-agent systems, and scalable multi-agent systems SE—5, vol 1887, pp 48–55. Springer, Berlin. doi:10.1007/3-540-47772-1_5

  • Habib-Agahi H, Ball G, Fox G (2009) NICM schedule & cost rules of thumb. In: AIAA space 2009 Conference & Exposition. AIAA, Pasadena, pp 6512–6512

  • Hart P, Duda R, Einaudi M (1978) PROSPECTOR—a computer-based consultation system for mineral exploration. Math Geol (November 1977). Retrieved from http://www.springerlink.com/index/3520V0M3W1864773.pdf

  • Haskins C (2006) INCOSE systems engineering handbook—a guide for system life cycle processes and activities (No. INCOSE-TP-2003-002-03). International Council on Systems Engineering

  • Hauser J, Clausing D (1988) The house of quality. Harv Bus Rev (May–June 1998)

  • Heliere F, Lin CC, Fois F, Davidson M, Thompson A, Bensi P (2009) BIOMASS: a P-band SAR earth explorer core mission candidate. In: Proceedings of the 2009 IEEE radar conference. IEEE. doi:10.1109/RADAR.2009.4977088

  • Ignizio J (1983) Generalized goal programming—an overview. Comput Operat Res 10(4):277–289

  • Keeney RL, Raiffa H (1976) Decisions with multiple objectives: preferences and value trade-offs. Wiley, New York, p 592

    Google Scholar 

  • Koo BHY, Simmons WL, Crawley EF (2009) Algebra of systems: a metalanguage for model synthesis and evaluation. IEEE Trans Syst Man Cybern Part A Syst Hum 39(3):501–513. doi:10.1109/TSMCA.2009.2014546

    Article  Google Scholar 

  • Kurtoglu T, Campbell MI (2009) An evaluation scheme for assessing the worth of automatically generated design alternatives. Res Eng Design 20(1):59–76. doi:10.1007/s00163-008-0062-1

    Article  Google Scholar 

  • Lai YJ, Liu TY, Hwang CL (1994) Topsis for MODM. Eur J Oper Res 76(3):486–500

    Article  MATH  Google Scholar 

  • Liao S (2005) Expert system methodologies and applications—a decade review from 1995 to 2004. Expert Syst Appl 28(1):93–103. doi:10.1016/j.eswa.2004.08.003

    Article  Google Scholar 

  • Lighthill J (1973) Artificial Intelligence: a general survey. In: B. S. R. Council (ed) Artificial intelligence: a paper symposium

  • Lindsay R, Buchanan BG, Feigenbaum EA (1993) DENDRAL: a case study of the first expert system for scientific hypothesis formation. Artif Intell 61(2):209–261. doi:10.1016/0004-3702(93)90068-M

    Article  Google Scholar 

  • Malen DE, Hancock WM (1995) Engineering for the customer: combining preference and physical systems models: part I—theory. J Eng Des 6(4):315–328

    Article  Google Scholar 

  • Mattson CA, Messac A (2003) Concept selection using s-Pareto frontiers. AIAA J 41(6):1190–1198. doi:10.2514/2.2063

    Article  Google Scholar 

  • Mauchand M, Siadat A, Bernard A, Perry N (2008) Proposal for tool-based method of product cost estimation during conceptual design. J Eng Des 19(2):159–172. doi:10.1080/09544820701802857

    Article  Google Scholar 

  • McDermott J (1982) R1: a rule-based configurer of computer systems. Artif Intell 19(1):39–88. doi:10.1016/0004-3702(82)90021-2

    Article  Google Scholar 

  • Messac A, Ismail-Yahaya A (2002) Multiobjective robust design using physical programming. Struct Multidiscip Optim 23(5):357–371. doi:10.1007/s00158-002-0196-0

    Article  Google Scholar 

  • Messac A, Ismail-Yahaya A, Mattson CA (2003) The normalized normal constraint method for generating the Pareto frontier. Struct Multidiscip Optim 25(2):86–98. doi:10.1007/s00158-002-0276-1

    Article  MathSciNet  MATH  Google Scholar 

  • Minsky M (1975) A framework for representing knowledge. In: Whinston PH (ed) The psychology of computer vision. McGraw-Hill Book, New York, pp 1–81

    Google Scholar 

  • Mon DL, Cheng CH, Lin JC (1994) Evaluating weapon system using fuzzy analytic hierarchy process based on entropy weight. Fuzzy Sets Syst 62(2):127–134

    Article  Google Scholar 

  • Newell A, Simon HA (1972) Human problem solving. Prentice Hall, Englewood Cliffs

    Google Scholar 

  • Nii H (1986) The blackboard model of problem solving and the evolution of blackboard architectures. AI Magazine 7(2):38–53. Retrieved from http://www.aaai.org/ojs/index.php/aimagazine/article/viewArticle/537

  • Parasuraman R, Sheridan TB, Wickens CD (2000) A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern Part A Syst Hum 30(3):286–297

    Article  Google Scholar 

  • Pareto V (1896) Cours d’economie politique. Librairie Droz, Geneva

    Google Scholar 

  • Park J, Han SH (2004) A fuzzy rule-based approach to modeling affective user satisfaction towards office chair design. Int J Ind Ergon 34(1):31–47

    Article  Google Scholar 

  • Poundstone W (1984) The recursive universe: cosmic complexity and the limits of scientific knowledge. William Morrow, New York, pp 1–252

  • Pugh S (1991) Total design: integrated methods for successful product engineering. Addison Wesley Publishing Company, Workingham

    Google Scholar 

  • Purves D (2010) Brains: how they seem to work. FT Press Science, Upper Saddle River, New Jersey, pp 1–320

  • Radovcic Y, Remouchamps A (2002) BOSS QUATTRO: an open system for parametric design. Struct Multidiscip Optim 23(2):140–152. doi:10.1007/s00158-002-0173-7

    Article  Google Scholar 

  • Reich Y (2010) My method is better! Res Eng Design 21(3):137–142. doi:10.1007/s00163-010-0092-3

    Article  Google Scholar 

  • Rohl PJ, Kolonay RM, Irani RK, Sobolewski M, Kao K, Bailey MW (2000) A federated intelligent product environment. In: 8th Symposium on multidisciplinary analysis and optimization

  • Ross AM, Hastings DE, Warmkessel JM, Diller NP (2004) Multi-attribute tradespace exploration as front end for effective space system design. J Spacecr Rocket 41(1):20–28

    Article  Google Scholar 

  • Saaty TL (1977) A scaling method for priorities in hierarchical structures. J Math Psychol 15(3):234–281. doi:10.1016/0022-2496(77)90033-5

    Article  MathSciNet  MATH  Google Scholar 

  • Saaty TL (1990) How to make a decision: the analytic hierarchy process. Eur J Oper Res 48(1):9–26. doi:10.1016/0377-2217(90)90057-I

    Article  MATH  Google Scholar 

  • Salomon S, Dom C, Avigad G, Freitas A, Goldvard A, Sch O (2014) PSA based multi objective evolutionary algorithms. In: Schuetze O, Coello Coello CA, Tantar A–A, Tantar E, Bouvry P, Del Moral P, Legrand P (eds) EVOLVE—a bridge between probability, set oriented numerics, and evolutionary computation III. Springer, Heidelberg, pp 233–255. doi:10.1007/978-3-319-01460-9

  • Schreiber G, Akkermans H, Anjewierden A, de Hoog R, Shadbolt N, Van De Velde W, Wielinga B (2000) Knowledge engineering and management: the CommonKADS methodology. MIT Press, Cambridge

    Google Scholar 

  • Seher T (2009) Campaign-level science traceability for earth observation system architecting (MS Thesis). Dept. of Aeronautics and Astronautics. Retrieved from http://dspace.mit.edu/handle/1721.1/51639?show=full

  • Selva D (2012) Rule-based system architecting of Earth observation satellite systems (PhD dissertation, Massachusetts Institute of Technology). ProQuest/UMI, Ann Arbor

  • Selva D, Crawley E (2013) VASSAR: value assessment of system architectures using rules. In: Aerospace conference, 2013 IEEE. Big Sky: IEEE

  • Shah JJ, Rogers MT (1993) Assembly modeling as an extension of feature-based design. Res Eng Design 5(3):218–237

    Article  Google Scholar 

  • Smith P, Reinertsen D (1997) Developing products in half the time: new rules, new tools, 2nd edn. Wiley, London

    Google Scholar 

  • Spanoudakis G, Zisman A, Pérez-Miñana E, Krause P (2004) Rule-based generation of requirements traceability relations. J Syst Softw 72(2):105–127. doi:10.1016/S0164-1212(03)00242-5

    Article  Google Scholar 

  • Stewart T (1992) A critical survey on the status of multiple criteria decision making theory and practice. Omega 20(5–6):569–586. doi:10.1016/0305-0483(92)90003-P

    Article  Google Scholar 

  • Stiny G (1980) Introduction to shape and shape grammars. Environ Plan 7(3):343–351. doi:10.1068/b070343

    Article  Google Scholar 

  • Suh N (1998) Axiomatic design theory for systems. Res Eng Design 10(4):189–209

    Article  Google Scholar 

  • Taguchi G, Elsayed E, Hsiang T (1989) Quality engineering in production systems. McGraw-Hill, New York

    Google Scholar 

  • Thurston DL (1991) A formal method for subjective design evaluation with multiple attributes. Res Eng Design 3(2):105–122. doi:10.1007/BF01581343

    Article  Google Scholar 

  • Ulrich K (1995) The role of product architecture in the manufacturing firm. Res Policy 24(3):419–440. doi:10.1016/0048-7333(94)00775-3

    Article  MathSciNet  Google Scholar 

  • Von Neumann J, Morgenstern O (1944) Theory of games and economic behavior. Princeton University Press, Princeton, p 625

    MATH  Google Scholar 

  • Welsch C, Swenson H (2001) VIIRS (Visible Infrared Imager Radiometer Suite): a next-generation operational environmental sensor for NPOESS. In: Proceedings of the 2001 international geoscience and remote sensing symposium, vol 3, pp 1020–1022

  • Wnuk K, Regnell B, Schrewelius C (2009) Architecting and coordinating thousands of requirements—an industrial case study. In: Glinz M, Heymans P (eds) Requirements engineering: foundation for software quality SE—10, vol 5512. Springer, Berlin, pp 118–123. doi:10.1007/978-3-642-02050-6_10

  • Wolfram S (2002) A new kind of science. Wolfram Media Inc, Champaign

    MATH  Google Scholar 

  • Yager RR (1977) Multiple objective decision-making using fuzzy sets. Int J Man Mach Stud 9(4):375–382

    Article  MathSciNet  MATH  Google Scholar 

  • Yager R (1988) On ordered weighted averaging aggregation operators in multicriteria decision making. Syst Man Cybern IEEE Trans 1:183–190

    Article  MathSciNet  Google Scholar 

  • Zadeh L (1963) Optimality and non-scalar-valued performance criteria. IEEE Trans Autom Control 8(1):59–60

  • Zadeh LA (1965) Fuzzy Sets. Inf Control 8(3):338–353. doi:10.1016/0165-0114(78)90029-5

    Article  MathSciNet  MATH  Google Scholar 

  • Zimmerman HJ (1983) Using fuzzy sets in operational research. Eur J Oper Res 13(3):201–216

    Article  MathSciNet  Google Scholar 

  • Ziv-Av A, Reich Y (2005) SOS—subjective objective system for generating optimal product concepts. Des Stud 26(5):509–533. doi:10.1016/j.destud.2004.12.001

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Sibley School of Mechanical and Aerospace Engineering, Cornell University, 212 Upson Hall, Ithaca, NY, 14850, USA

    Daniel Selva

  2. Department of Aeronautics and Astronautics, MIT, 77 Massachusetts Ave Room 33-409, Cambridge, MA, 02139, USA

    Daniel Selva

  3. Engineering Systems Division, MIT, 77 Massachusetts Ave Room 31-161b, Cambridge, MA, 02139, USA

    Bruce Cameron

  4. Department of Aeronautics and Astronautics, MIT, 77 Massachusetts Ave Room 33-413, Cambridge, MA, 02139, USA

    Edward F. Crawley

Authors
  1. Daniel Selva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruce Cameron
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Edward F. Crawley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Selva.

Appendices

Appendix 1: Hierarchy of stakeholder needs for Earth observation example

The information used in the case study for the stakeholders and their hierarchy of requirements are shown below in Tables 6 and 7.

Table 6 Stakeholders and weights
Full size table
Table 7 Stakeholder objectives and weights
Full size table

Appendix 2: Characteristics and capabilities of instruments for Earth observation example

The characteristics of the instruments are provided in the Tables 8 and 9.

Table 8 Instrument characteristics
Full size table
Table 9 Instrument capabilities
Full size table

Appendix 3: Cost model

The cost model used in the case study is a rule-based cost model largely based on Larson and Wertz’s Space Mission Analysis and Design. The first-level decomposition of lifecycle cost is given in Fig. 15.

Fig. 15
figure 15

Lifecycle cost decomposition

Full size image

Payload cost is based on the NASA Instrument Cost model (Habib-Agahi et al. 2009). Bus cost is based on the parametric provided in Apgar (2011). Since these parametrics are based on the spacecraft mass budget, a spacecraft design module that estimates the mass and power budgets of each spacecraft precedes the cost estimation module.

The spacecraft design module is iterative because of the couplings between different subsystems. For example, the mass of the spacecraft affects the design of the ADCS through the size of the reaction wheels and the amount of propellant among others, and these feed back into the computation of the spacecraft mass. In practice, three iterations are sufficient to make the design process converge to a precision of less than a kg. An overview of the spacecraft design module is provided in Fig. 16.

Fig. 16
figure 16

Spacecraft design algorithm

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Selva, D., Cameron, B. & Crawley, E.F. A rule-based method for scalable and traceable evaluation of system architectures. Res Eng Design 25, 325–349 (2014). https://doi.org/10.1007/s00163-014-0180-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00163-014-0180-x

Keywords

Search

Navigation