Research in Engineering Design

, Volume 25, Issue 4, pp 325–349 | Cite as

A rule-based method for scalable and traceable evaluation of system architectures

  • Daniel SelvaEmail author
  • Bruce Cameron
  • Edward F. Crawley
Original Paper


Despite the development of a variety of decision-aid tools for assessing the value of a conceptual design, humans continue to play a dominant role in this process. Researchers have identified two major challenges to automation, namely the subjectivity of value and the existence of multiple and conflicting customer needs. A third challenge is however arising as the amount of data (e.g., expert judgment, requirements, and engineering models) required to assess value increases. This brings two challenges. First, it becomes harder to modify existing knowledge or add new knowledge into the knowledge base. Second, it becomes harder to trace the results provided by the tool back to the design variables and model parameters. Current tools lack the scalability and traceability required to tackle these knowledge-intensive design evaluation problems. This work proposes a traceable and scalable rule-based architecture evaluation tool called VASSAR that is especially tailored to tackle knowledge-intensive problems that can be formulated as configuration design problems, which is demonstrated using the conceptual design task for a laptop. The methodology has three main steps. First, facts containing the capabilities and performance of different architectures are computed using rules containing physical and logical models. Second, capabilities are compared with requirements to assess satisfaction of each requirement. Third, requirement satisfaction is aggregated to yield a manageable number of metrics. An explanation facility keeps track of the value chain all along this process. This paper describes the methodology in detail and discusses in particular different implementations of preference functions as logical rules. A full-scale example around the design of Earth observing satellites is presented.


Conceptual design Design evaluation Requirement traceability Rule-based systems 


  1. Antonsson EK, Cagan J (2001) Formal engineering design synthesis. Cambridge University Press, Cambridge. doi: 10.1017/CBO9780511529627 CrossRefGoogle Scholar
  2. Apgar H (2011) Cost estimating. In: Wertz JR, Everett DF, Puschell JJ (eds) Space mission engineering: the new SMAD. Microcosm, HawthorneGoogle Scholar
  3. Armacost RL, Componation PJ, Mullens MA, Swart WS (1994) An AHP framework for prioritizing customer requirements in QFD: an industrialized housing application. IIE Trans 26(4):72–79CrossRefGoogle Scholar
  4. Avigad G, Moshaiov A (2009) Interactive evolutionary multiobjective search and optimization of set-based concepts. IEEE Trans Syst Man Cybern Part B Cybern 39(4):1013–1027. doi: 10.1109/TSMCB.2008.2011565 CrossRefGoogle Scholar
  5. Bellman RE, Zadeh LA (1970) Decision-making in a fuzzy environment. Manag Sci 17(4):B-141MathSciNetCrossRefGoogle Scholar
  6. Bonczek RH, Holsapple CW, Whinston AB (1981) A generalized decision support system using predicate calculus and network data base management. Oper Res 29(2):263–281. doi: 10.2307/170020 MathSciNetCrossRefGoogle Scholar
  7. Buchanan BG, Shortliffe EH (1984) Rule-based expert systems: the MYCIN experiments of the Stanford Heuristic Programming Project. Language. Addison-Wesley, ReadingGoogle Scholar
  8. Cameron B (2008) Value flow mapping: using networks to inform stakeholder analysis. Acta Astronaut 62(4–5):324–333. doi: 10.1016/j.actaastro.2007.10.001 CrossRefGoogle Scholar
  9. Campbell M, Cagan J, Kotovsky K (1999) A-design: an agent-based approach to conceptual design in a dynamic environment. Res Eng Design 11:172–192Google Scholar
  10. Carlson-Skalak S, White M, Teng Y (1998) Using an evolutionary algorithm for catalog design. Res Eng Design 10:63–83Google Scholar
  11. Chalmers University of Technology (2004) Use of P-band SAR for forest biomass and soil moisture retrieval. Retrieved from
  12. Chandrasekaran B (1989) A framework for design problem-solving. Res Eng Design 1:75–86CrossRefGoogle Scholar
  13. Charnes A, Cooper WW (1957) Management models and industrial applications of linear programming. Manage Sci 4(1):38–91MathSciNetzbMATHCrossRefGoogle Scholar
  14. Clancey WJ (1987) Knowledge-based tutoring: the GUIDON program. In: MIT Press series in artificial intelligence. The MIT Press, CambridgeGoogle Scholar
  15. Corkill DD (2003) Blackboard and multi-agent system & the future. In: Proceedings of the international lisp conference, New York, NY. Retrieved from
  16. Crawley E, De Weck O, Eppinger S, Magee C, Moses J, Seering W et al (2004) The influence of architecture in engineering systems. In: Engineering systems monograph. ArchitectureGoogle Scholar
  17. Das I, Dennis JE (1998) Normal-boundary intersection: a new method for generating the pareto surface in nonlinear multicriteria optimization problems. SIAM J Optim 8(3). doi: 10.1137/S1052623496307510
  18. De Condorcet M (1785) Essai sur l’application de l’analyse à la probabilité des décisions rendues à la probabilité des voix. De l’Imprimerie Royale, ParisGoogle Scholar
  19. De Weck OL, Kim IY (2004) Adaptive weighted sum method for bi-objective optimization. In: Proceedings of the 45th AIAA/ASME/ASCE/, pp 1–13Google Scholar
  20. Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6(2):182–197. doi: 10.1109/4235.996017 CrossRefGoogle Scholar
  21. Dincbas M (1980) A knowledge-based expert system for automatic analysis and synthesis in CAD. In: IFIP congress (pp 705–710)Google Scholar
  22. Dobias AP (1990) Designing a mouse trap using the analytic hierarchy process and expert choice. Eur J Oper Res 48(1):57–65CrossRefGoogle Scholar
  23. Donaldson KM, Ishii K, Sheppard SD (2006) Customer value chain analysis. Res Eng Design 16(4):174–183. doi: 10.1007/s00163-006-0012-8 CrossRefGoogle Scholar
  24. Dori D (2002) Object-process methodology: a holistic paradigm. Springer, Berlin, pp 1–453CrossRefGoogle Scholar
  25. Duda RO, Gaschnig JG, Hart PE (1979) Model design in the PROSPECTOR consultant system for mineral exploration. In: Michie D (ed) Expert systems in the microelectronic age. Edinburgh University Press, Edinburgh, pp 153–167Google Scholar
  26. Durkin J (1990) Application of expert systems in the sciences. Ohio J Sci 90(5):171–179Google Scholar
  27. Engelmore R, Morgan T (eds) (1988) Blackboard systems. Addison-Wesley Publishing Company, Reading, pp 1–620Google Scholar
  28. Entekhabi D (2010) The soil moisture active passive (SMAP) mission. In: Proceedings of the IEEE, vol 98, pp 704–716. doi: 10.1109/JPROC.2010.2043918
  29. Erman L, Hayes-Roth F (1980) The Hearsay-II speech-understanding system: integrating knowledge to resolve uncertainty. ACM Comput Surv 12(2):213–253.
  30. Feigenbaum EA, Buchanan BG, Lederberg J (1971) On generality and problem solving: a case study using the DENDRAL program. In: Meltzer B, Michie D (eds) Machine Intelligence 6. Edinburgh, Scotland, pp 165–190Google Scholar
  31. Fenves S, Garrett J (1986) Knowledge based standards processing. Artif Intell Eng 1(1):3–14. doi: 10.1016/0954-1810(86)90029-4 CrossRefGoogle Scholar
  32. Fishburn PC (1974) Lexicographic orders, utilities and decision rules: a survey. Manag Sci 20(11):1442–1471Google Scholar
  33. Fortin J, Dubois D, Fargier H (2008) Gradual numbers and their application to fuzzy interval analysis. IEEE Trans Fuzzy Syst 16(2):388–402. doi: 10.1109/TFUZZ.2006.890680 CrossRefGoogle Scholar
  34. Gasster S, Flaming GGM (1998) Overview of the conical microwave imager/sounder development for the NPOESS program. In: Geoscience and remote sensing …, vol 1, pp 268–270. IEEE. doi: 10.1109/IGARSS.1998.702874
  35. Geoffrion AM, Dyer JS, Feinberg A (1972) An interactive approach for multi-criterion optimization, with an application to the operation of an academic department. Manag Sci 19(4):357–368Google Scholar
  36. Giarratano JC, Riley GD (2004) Expert systems: principles and programming, 4th edn. Course Technology. Retrieved from
  37. Gologlu C, Mizrak C (2011) An integrated fuzzy logic approach to customer-oriented product design. J Eng Des 22(2):113–127. doi: 10.1080/09544820903032519 CrossRefGoogle Scholar
  38. Gutknecht O, Ferber J (2001) The MadKit agent platform architecture. In: Wagner T, Rana O (eds) Infrastructure for agents, multi-agent systems, and scalable multi-agent systems SE—5, vol 1887, pp 48–55. Springer, Berlin. doi: 10.1007/3-540-47772-1_5
  39. Habib-Agahi H, Ball G, Fox G (2009) NICM schedule & cost rules of thumb. In: AIAA space 2009 Conference & Exposition. AIAA, Pasadena, pp 6512–6512Google Scholar
  40. Hart P, Duda R, Einaudi M (1978) PROSPECTOR—a computer-based consultation system for mineral exploration. Math Geol (November 1977). Retrieved from
  41. Haskins C (2006) INCOSE systems engineering handbook—a guide for system life cycle processes and activities (No. INCOSE-TP-2003-002-03). International Council on Systems EngineeringGoogle Scholar
  42. Hauser J, Clausing D (1988) The house of quality. Harv Bus Rev (May–June 1998)Google Scholar
  43. Heliere F, Lin CC, Fois F, Davidson M, Thompson A, Bensi P (2009) BIOMASS: a P-band SAR earth explorer core mission candidate. In: Proceedings of the 2009 IEEE radar conference. IEEE. doi: 10.1109/RADAR.2009.4977088
  44. Ignizio J (1983) Generalized goal programming—an overview. Comput Operat Res 10(4):277–289Google Scholar
  45. Keeney RL, Raiffa H (1976) Decisions with multiple objectives: preferences and value trade-offs. Wiley, New York, p 592Google Scholar
  46. Koo BHY, Simmons WL, Crawley EF (2009) Algebra of systems: a metalanguage for model synthesis and evaluation. IEEE Trans Syst Man Cybern Part A Syst Hum 39(3):501–513. doi: 10.1109/TSMCA.2009.2014546 CrossRefGoogle Scholar
  47. Kurtoglu T, Campbell MI (2009) An evaluation scheme for assessing the worth of automatically generated design alternatives. Res Eng Design 20(1):59–76. doi: 10.1007/s00163-008-0062-1 CrossRefGoogle Scholar
  48. Lai YJ, Liu TY, Hwang CL (1994) Topsis for MODM. Eur J Oper Res 76(3):486–500zbMATHCrossRefGoogle Scholar
  49. Liao S (2005) Expert system methodologies and applications—a decade review from 1995 to 2004. Expert Syst Appl 28(1):93–103. doi: 10.1016/j.eswa.2004.08.003 CrossRefGoogle Scholar
  50. Lighthill J (1973) Artificial Intelligence: a general survey. In: B. S. R. Council (ed) Artificial intelligence: a paper symposiumGoogle Scholar
  51. Lindsay R, Buchanan BG, Feigenbaum EA (1993) DENDRAL: a case study of the first expert system for scientific hypothesis formation. Artif Intell 61(2):209–261. doi: 10.1016/0004-3702(93)90068-M CrossRefGoogle Scholar
  52. Malen DE, Hancock WM (1995) Engineering for the customer: combining preference and physical systems models: part I—theory. J Eng Des 6(4):315–328CrossRefGoogle Scholar
  53. Mattson CA, Messac A (2003) Concept selection using s-Pareto frontiers. AIAA J 41(6):1190–1198. doi: 10.2514/2.2063 CrossRefGoogle Scholar
  54. Mauchand M, Siadat A, Bernard A, Perry N (2008) Proposal for tool-based method of product cost estimation during conceptual design. J Eng Des 19(2):159–172. doi: 10.1080/09544820701802857 CrossRefGoogle Scholar
  55. McDermott J (1982) R1: a rule-based configurer of computer systems. Artif Intell 19(1):39–88. doi: 10.1016/0004-3702(82)90021-2 CrossRefGoogle Scholar
  56. Messac A, Ismail-Yahaya A (2002) Multiobjective robust design using physical programming. Struct Multidiscip Optim 23(5):357–371. doi: 10.1007/s00158-002-0196-0 CrossRefGoogle Scholar
  57. Messac A, Ismail-Yahaya A, Mattson CA (2003) The normalized normal constraint method for generating the Pareto frontier. Struct Multidiscip Optim 25(2):86–98. doi: 10.1007/s00158-002-0276-1 MathSciNetzbMATHCrossRefGoogle Scholar
  58. Minsky M (1975) A framework for representing knowledge. In: Whinston PH (ed) The psychology of computer vision. McGraw-Hill Book, New York, pp 1–81Google Scholar
  59. Mon DL, Cheng CH, Lin JC (1994) Evaluating weapon system using fuzzy analytic hierarchy process based on entropy weight. Fuzzy Sets Syst 62(2):127–134CrossRefGoogle Scholar
  60. Newell A, Simon HA (1972) Human problem solving. Prentice Hall, Englewood CliffsGoogle Scholar
  61. Nii H (1986) The blackboard model of problem solving and the evolution of blackboard architectures. AI Magazine 7(2):38–53. Retrieved from
  62. Parasuraman R, Sheridan TB, Wickens CD (2000) A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern Part A Syst Hum 30(3):286–297CrossRefGoogle Scholar
  63. Pareto V (1896) Cours d’economie politique. Librairie Droz, GenevaGoogle Scholar
  64. Park J, Han SH (2004) A fuzzy rule-based approach to modeling affective user satisfaction towards office chair design. Int J Ind Ergon 34(1):31–47CrossRefGoogle Scholar
  65. Poundstone W (1984) The recursive universe: cosmic complexity and the limits of scientific knowledge. William Morrow, New York, pp 1–252Google Scholar
  66. Pugh S (1991) Total design: integrated methods for successful product engineering. Addison Wesley Publishing Company, WorkinghamGoogle Scholar
  67. Purves D (2010) Brains: how they seem to work. FT Press Science, Upper Saddle River, New Jersey, pp 1–320Google Scholar
  68. Radovcic Y, Remouchamps A (2002) BOSS QUATTRO: an open system for parametric design. Struct Multidiscip Optim 23(2):140–152. doi: 10.1007/s00158-002-0173-7 CrossRefGoogle Scholar
  69. Reich Y (2010) My method is better! Res Eng Design 21(3):137–142. doi: 10.1007/s00163-010-0092-3 CrossRefGoogle Scholar
  70. Rohl PJ, Kolonay RM, Irani RK, Sobolewski M, Kao K, Bailey MW (2000) A federated intelligent product environment. In: 8th Symposium on multidisciplinary analysis and optimizationGoogle Scholar
  71. Ross AM, Hastings DE, Warmkessel JM, Diller NP (2004) Multi-attribute tradespace exploration as front end for effective space system design. J Spacecr Rocket 41(1):20–28CrossRefGoogle Scholar
  72. Saaty TL (1977) A scaling method for priorities in hierarchical structures. J Math Psychol 15(3):234–281. doi: 10.1016/0022-2496(77)90033-5 MathSciNetzbMATHCrossRefGoogle Scholar
  73. Saaty TL (1990) How to make a decision: the analytic hierarchy process. Eur J Oper Res 48(1):9–26. doi: 10.1016/0377-2217(90)90057-I zbMATHCrossRefGoogle Scholar
  74. Salomon S, Dom C, Avigad G, Freitas A, Goldvard A, Sch O (2014) PSA based multi objective evolutionary algorithms. In: Schuetze O, Coello Coello CA, Tantar A–A, Tantar E, Bouvry P, Del Moral P, Legrand P (eds) EVOLVE—a bridge between probability, set oriented numerics, and evolutionary computation III. Springer, Heidelberg, pp 233–255. doi: 10.1007/978-3-319-01460-9
  75. Schreiber G, Akkermans H, Anjewierden A, de Hoog R, Shadbolt N, Van De Velde W, Wielinga B (2000) Knowledge engineering and management: the CommonKADS methodology. MIT Press, CambridgeGoogle Scholar
  76. Seher T (2009) Campaign-level science traceability for earth observation system architecting (MS Thesis). Dept. of Aeronautics and Astronautics. Retrieved from
  77. Selva D (2012) Rule-based system architecting of Earth observation satellite systems (PhD dissertation, Massachusetts Institute of Technology). ProQuest/UMI, Ann ArborGoogle Scholar
  78. Selva D, Crawley E (2013) VASSAR: value assessment of system architectures using rules. In: Aerospace conference, 2013 IEEE. Big Sky: IEEEGoogle Scholar
  79. Shah JJ, Rogers MT (1993) Assembly modeling as an extension of feature-based design. Res Eng Design 5(3):218–237CrossRefGoogle Scholar
  80. Smith P, Reinertsen D (1997) Developing products in half the time: new rules, new tools, 2nd edn. Wiley, LondonGoogle Scholar
  81. Spanoudakis G, Zisman A, Pérez-Miñana E, Krause P (2004) Rule-based generation of requirements traceability relations. J Syst Softw 72(2):105–127. doi: 10.1016/S0164-1212(03)00242-5 CrossRefGoogle Scholar
  82. Stewart T (1992) A critical survey on the status of multiple criteria decision making theory and practice. Omega 20(5–6):569–586. doi: 10.1016/0305-0483(92)90003-P CrossRefGoogle Scholar
  83. Stiny G (1980) Introduction to shape and shape grammars. Environ Plan 7(3):343–351. doi: 10.1068/b070343 CrossRefGoogle Scholar
  84. Suh N (1998) Axiomatic design theory for systems. Res Eng Design 10(4):189–209CrossRefGoogle Scholar
  85. Taguchi G, Elsayed E, Hsiang T (1989) Quality engineering in production systems. McGraw-Hill, New YorkGoogle Scholar
  86. Thurston DL (1991) A formal method for subjective design evaluation with multiple attributes. Res Eng Design 3(2):105–122. doi: 10.1007/BF01581343 CrossRefGoogle Scholar
  87. Ulrich K (1995) The role of product architecture in the manufacturing firm. Res Policy 24(3):419–440. doi: 10.1016/0048-7333(94)00775-3 MathSciNetCrossRefGoogle Scholar
  88. Von Neumann J, Morgenstern O (1944) Theory of games and economic behavior. Princeton University Press, Princeton, p 625zbMATHGoogle Scholar
  89. Welsch C, Swenson H (2001) VIIRS (Visible Infrared Imager Radiometer Suite): a next-generation operational environmental sensor for NPOESS. In: Proceedings of the 2001 international geoscience and remote sensing symposium, vol 3, pp 1020–1022Google Scholar
  90. Wnuk K, Regnell B, Schrewelius C (2009) Architecting and coordinating thousands of requirements—an industrial case study. In: Glinz M, Heymans P (eds) Requirements engineering: foundation for software quality SE—10, vol 5512. Springer, Berlin, pp 118–123. doi: 10.1007/978-3-642-02050-6_10
  91. Wolfram S (2002) A new kind of science. Wolfram Media Inc, ChampaignzbMATHGoogle Scholar
  92. Yager RR (1977) Multiple objective decision-making using fuzzy sets. Int J Man Mach Stud 9(4):375–382MathSciNetzbMATHCrossRefGoogle Scholar
  93. Yager R (1988) On ordered weighted averaging aggregation operators in multicriteria decision making. Syst Man Cybern IEEE Trans 1:183–190MathSciNetCrossRefGoogle Scholar
  94. Zadeh L (1963) Optimality and non-scalar-valued performance criteria. IEEE Trans Autom Control 8(1):59–60Google Scholar
  95. Zadeh LA (1965) Fuzzy Sets. Inf Control 8(3):338–353. doi: 10.1016/0165-0114(78)90029-5 MathSciNetzbMATHCrossRefGoogle Scholar
  96. Zimmerman HJ (1983) Using fuzzy sets in operational research. Eur J Oper Res 13(3):201–216MathSciNetCrossRefGoogle Scholar
  97. Ziv-Av A, Reich Y (2005) SOS—subjective objective system for generating optimal product concepts. Des Stud 26(5):509–533. doi: 10.1016/j.destud.2004.12.001 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • Daniel Selva
    • 1
    • 2
    Email author
  • Bruce Cameron
    • 3
  • Edward F. Crawley
    • 4
  1. 1.Sibley School of Mechanical and Aerospace EngineeringCornell UniversityIthacaUSA
  2. 2.Department of Aeronautics and AstronauticsMITCambridgeUSA
  3. 3.Engineering Systems DivisionMITCambridgeUSA
  4. 4.Department of Aeronautics and AstronauticsMITCambridgeUSA

Personalised recommendations