Skip to main content
SpringerLink
Log in

Quality Assurance in Modelling and Simulation: A Taxonomy

  • Conference paper
Simulation and Model-Based Methodologies: An Integrative View

Part of the book series: NATO ASI Series ((NATO ASI F,volume 10))

Abstract

A basis for the taxonomy of concepts related with quality assurance in modelling and simulation is provided. It has three parts:

  1. 1)

    Basic elements of modelling and simulation which could and should be assessed

  2. 2)

    Criteria which can be used for assessment, and

  3. 3)

    Types of assessments.

The comprehensive categorized list of related terms given can be the starting point

  1. 1)

    for a systematic study of the possibilities

  2. 2)

    for their systematic refinements

  3. 3)

    for the development of systematic studies of quality assurance in modelling and simulation, and

  4. 4)

    for compiling a glossary of terms of quality assurance in modelling and simulation.

This study has been sponsored by the Natural Sciences and Engineering Research Council of Canada (NSERC) by the Operating Grant A8117.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References and Bibliography

  • Allen, B., Clema, J.K. (1976). Independent Verification/Validation Support Software. In: Proceedings of the Aerospace and Electronics Conference, Dayton, Ohio, pp. 276–281.

    Google Scholar 

  • Annino, J.S., Russell, E.C. (1979). The ten Most Frequent Causes of Simulation Analysis Failure–And How to Avoid Them. Simulation Today, Vol. 32, No. 60, pp. 137–140.

    Article  Google Scholar 

  • ANSI. Quality Assurance Terms and Definitions. Publication No. N45.2.10–1973. American National Standards Committee N45, American Society of Mechanical Engineers, New York.

    Google Scholar 

  • ANSI/ASQC. Quality Systems Terminology. Publication No. A3–1078. American Society for Quality Control, Milwaukee, Wisconsin.

    Google Scholar 

  • Bahr, D (1975). A Note on Figures of Merit. ACM Performance Evaluation Review, Vol. 4, No. 3, pp. 1–3.

    Article  Google Scholar 

  • Balci, O. (1981). Statistical Validation of Multivariate Response Simulation Models. Ph.D. Dissertation, Graduate School of Syracuse University, Syracuse, NY.

    Google Scholar 

  • Balci, O., Sargent, R.G. 1980a. A Methodology for Cost-Risk Analysis in Validation of Simulation Models. Working Paper No. 80–006, Department of Industrial Engineering and Operations Research, Syracuse University, Syracuse, NY.

    Google Scholar 

  • Balci, O., Sargent, R.G. 1980. Bibliography on Validation of Simulation Models. Newsletter, TIMS College on Simulation and Gaming, Vol. 4, No. 4, Spring 1980, pp. 11–15 (125 entries).

    Google Scholar 

  • Balci, O. Sargent, R.G. 1981a. A Methodology for Validating Multivariate Response Simulation Models by Using Simultaneous Confidence Intervals. Technical Report CS81019-R, Computer Science Department, Virginia Polytechnic Institute and State University, Blacksburg, VA.

    Google Scholar 

  • Balci, O., Sargent 1981b. Validation of Multivariate Response Simulation Models By Using Hotelling’s Two-Sample t2 Test. Technical Report CS81016-R, Computer Science Department, Virginia Polytechnic Institute and State University, Blacksburg, VA.

    Google Scholar 

  • Balci, O., Sargent, R.G. (1984). A Bibliography on the Credibility Assessment and Validation of Simulation and Mathematical Models. Simuletter, Vol. 15, No. 3.

    Google Scholar 

  • Belady, L.A., Leavenworth, B. (1980). Program Modifiability. Report RC 8147, IBM T.J. Watson Research Center, Yorktown Heights, NY.

    Google Scholar 

  • Berman, M.B. (1972). Notes on Validating/Verifying Computer Simulation Models. P-4891, Rand, Santa Monica, CA.

    Google Scholar 

  • Britting, K.R. (1976). Backward Integration of System Dynamics Models. In: World Modeling: A Dialogue, Churchman, C.W., Mason, R.O. (Eds.), North Holland, Amsterdam, p. 141–149.

    Google Scholar 

  • Cellier, F.E. (1979). Combined Continuous/Discrete System Simulation by Use of Digital Computers: Techniques and Tools. Dissertation, Doctor of Technical Sciences, ETH, Zurich, Switzerland.

    Google Scholar 

  • Cellier, F.E., Moebius, P.J. (1979). Towards Robust General Purpose Simulation Software. ACM SIGNUM.

    Google Scholar 

  • Chandy, K.M., Misra, J. (1980). On the Correctness of Simulation. (Pèrsonal Communication).

    Google Scholar 

  • Cutler, M.M. (1979) Proving Properties of Simulation Programs for System Verification and Validation. In: Proceedings of the 1979 Summer Computer Simulation Conference, Toronto, Ont., Canada, pp. 610–616.

    Google Scholar 

  • Cutler, M.M. (1980). A Formal Program Model for Discrete Event Simulation and Its Use in the Verification and Validation of System Models and Implementations. Ph.D. Dissertation, University of California, Los Angeles, CA

    Google Scholar 

  • Deeter, C.R., Hoffman, A.A.J. (1978). An Annotated Bibliography of Mathematical Models Classified by Validation and Error Analysis Methods. Texas, Christian University, Fort Worth, Texas, 112 p.

    Google Scholar 

  • Dickhoven, S. (1979). Software Requirements for an Improvement of Transfer and Adaptibility of Models. US National Bureau of Standards Workshop: ‘Utility and Use of Large-Scale Mathematical Models,’ Gaithersburg, MD.

    Google Scholar 

  • Drake, J.W. (1973). Why Models Fail in Transportation. In: Proceedings of 1973 Summer Computer Simulation Conference, pp. 1140–1144.

    Google Scholar 

  • Elzas, M.S. 1978a. Whither Simulation? In: Proceedings of the 1978 UKSC Conference on Computer Simulation, Chester, England 4–6 April, pp. 494–501.

    Google Scholar 

  • Elzas, M.S. 1978b. Results of a Simulation: Can They Be “Trusted”? In: Proceedings of the Seminar “Modelling in Business”, Amsterdam, Holland, 12–13 January, pp. 170–183.

    Google Scholar 

  • Elzas, M.S. (1979). What is Needed for Robust Simulation? In: Methodology in Systems Modelling and Simulation, B.P. Zeigler, M.S. Elzas, G.J. Klir, T.I. Oren (Eds.). North-Holland, Amsterdam, pp. 57–91.

    Google Scholar 

  • Ferrell, W.R., McGoey, P.J. (1980). A Model of Calibration for Subjective Probabilities. Organizational Behavior and Human Performance, Vol. 26, pp. 32–53.

    Article  Google Scholar 

  • Gass, S.I. 1977a. Evaluation of Complex Models. Computers and Operations Research, Vol. 4, No. 1, pp. 27–35.

    Article  Google Scholar 

  • Gass, S.I. 1977b. A Procedure for the Evaluation of Complex Models. In: Proceedings of the First International Conference in Mathematical Models. University of Missouri.

    Google Scholar 

  • Gass, S.I. 1979a. Validation and Assessment Issues of Energy Models. Prepared for NATO Advanced Research Institute on “The Application of Systems Science to Energy Policy Planning,” November 12–16, Upton, NY.

    Google Scholar 

  • Gass, S.I. 1979b. Computer Science & Technology: Computer Model Documentation: A Review and an Approach. NBS Special Publication No. 500–39, National Bureau of Standards, Washington, DC.

    Google Scholar 

  • Gass, S.I. (Ed.) 1979c. Utility and Use of Large-Scale Mathematical Models. NBS Special Publication, No. 534, National Bureau of Standards, Washington, DC, 223 p.

    Google Scholar 

  • Gass, S.I. (Ed.) 1979d. Bibliography: Model Evaluation and Assessment. In: Utility and Use of Large-Scale Mathematical Models-Proceedings of a Workshop held at the National Bureau of Standards, Gaithersburg, Maryland, April 28–29, 1977, pp. 215–217.

    Google Scholar 

  • Gass, S.I. (Ed.) (1980). Validation and Assessment Issues of Energy Models. Proceedings of a Workshop Held at the National Bureau of Standards, Gaithersburg, Maryland, January 10–11. NBS Special Publication No. 569, National Bureau of Standards, Washington, DC, 570 p.

    Google Scholar 

  • Gass, S.I. (Ed.) 1981a. Validation and Assessment of Energy Models. NBS Special Publication No. 616, National Bureau of Standards, Washington, DC, 255 p.

    Google Scholar 

  • Gass, S.I. 1981b. Documentation for a Model: A Hierarchical Approach. Communications of the ACM, Vol. 24, No. 11, pp. 728–733.

    Article  Google Scholar 

  • Hailpern, B., Owicki, S. (1981). Modular Verification of Computer Communication Protocols. Report RC 8726, IBM T.J. Watson Research Center, Yorktown Heights, NY.

    Google Scholar 

  • Henize, J. (1979). Evaluating Policy Models. In: Proceedings of the Summer Computer Simulation Conference. Toronto, Ontario, Canada, pp 550–556.

    Google Scholar 

  • Herman, C. (1967). Validation Problems in Games and Simulations. Behavioral Science, Vol. 12, pp. 212–230.

    Article  Google Scholar 

  • Holling, C.S. (Ed.) (1978). Adaptive Environmental Assessment and Management. Wiley, New York, 377 pp.

    Google Scholar 

  • House, P.W. (1974). Diogenes Revisited — The Search for a Valid Model. Simulation, Vol. 23, No. 4, pp. 117–125.

    Article  Google Scholar 

  • Howden, W.E. (1980). Applicability of Software Validation Techniques to Scientific Programs. ACM Transactions on Programming Languages and Systems, Vol. 2, No. 3, pp. 307–320.

    Article  Google Scholar 

  • Hunt, B.R. (1976). Large Scale System Theories: Some Paradigms from Software Systems Engineering. In: Cybernetics and Modelling and Simulation of Large Scale Systems, T.I. Ören (Ed.), International Association for Cybernetics, Namur, Belgium.

    Google Scholar 

  • Kahne, S.E. (1976). Model Credibility for Large-Scale Systems. IEEE Transactions on Systems, Man, and Cybernetics, August 1976, pp. 586–590.

    Google Scholar 

  • Karr, M, Loveman T, (1978). Incorporation of Units into Programming Languages. Communications of the ACM, Vol. 21, No. 5, pp. 385–391.

    Article  Google Scholar 

  • Kheir, H.A., Holmes, W.M. (1978). On Validating Simulation Models of Missile Systems. Simulation, Vol. 30, No. 4, pp. 117–128.

    Article  Google Scholar 

  • Kleijnen, J.P.C. (1981). Cross-Validation Using the t Statistic. Research Memorandum, Tilburg University, Tilburg, Netherlands.

    Google Scholar 

  • Kobayashi, H. (1975). System Design and Performance Analysis Using Analytic Models. Report RA 75, IBM T.J. Watson Research Center, 92 pp.

    Google Scholar 

  • Landry, M., Malouin, J-L., Oral, M. (1983). Model Validation in Operations Research. European J. of Operational research, Vol. 14, pp. 207–220.

    Article  Google Scholar 

  • Lee jr., (1973). Requiem for Large-Scale Models, J. of American Institute of Planners, Vol. 39, No. 3.

    Google Scholar 

  • Lewandowski, A. (1982). Issues in Model Validation. RR-82–37, IIASA, Laxenburg, Austria. Reprinted from Angewandte Systemanalyse, Vol. 3, No. 1, 1982.

    Google Scholar 

  • Marmor-Squires, A.B. (1977). Software Validation: An Annotated Bibliography. In: Current Trends in Programming Methodology. Vol. II. Program Validation, R.T. Yeh (Ed.). Prentice-Hall, Englewood Cliffs, NY, pp. 269–319.

    Google Scholar 

  • McLeod, J. (1977). Terms and Definitions — Play it Again, Stew. Simulation Today, Vol. 29, No. 4, pp. vii-ix.

    Article  MathSciNet  Google Scholar 

  • McLeod, J. (1982). Computer Modeling and Simulation: Principles of Good Practice. Simulation Series, Vol. 10, No. 2, The Society for Computer Simulation, La Jolla, CA.

    Google Scholar 

  • Nance, R.E. (1977). The Feasibility of and Methodology for Developing Federal Documentation Standards for Simulation Models. Report, Dept. of Computer Science, VPI and State University Blacksburg, VA.

    Google Scholar 

  • Newton, O.L., Whetherbee, J.E. (1980). Guidelines for Documenting Simulation Models. In: Simulation with Discrete Models: A State-of-the-Art View, T.I. Ören, C.M. Shub, P.F. Roth (Eds.), IEEE, New York, pp. 243–258.

    Google Scholar 

  • Oren, T.I. (1979). Concepts for Advanced Computer-Assisted Modelling. In: Methodology in Systems Modelling and Simulation, B.P. Zeigler et al. (Eds.). North-Holland, pp. 29–55. (Opening Paper in the Symposium on Modelling and Simulation Methodology, Rehovot, Israel, 1978 August 13–18).

    Google Scholar 

  • Ören, T.I. 1981a. Foundations for an Information Technology. In: Proceedings of 1981 Winter Simulation Conference, T.I. Ören, C.M. Delfosse, C.M. Shub (Etls.). IEEE, New York, pp. 201–208.

    Google Scholar 

  • Ören, T.I. 1981b. Concepts and Criteria to Assess Acceptability of Simulation Studies: A Frame of Reference. CACM, Vol. 24, No. 4, pp. 180–189.

    Google Scholar 

  • Ören, T.I. 1982a. Computer-Aided Modelling Systems. (Keynote paper — SIMULATION ‘80, Interlaken, Switzerland, 1980 June 25–27), In: Progress in Modelling and Simulation, F.E. Cellier (Ed.). Academic Press, London, England, pp. 189–203.

    Google Scholar 

  • Ören, T.I. 1982b. Synergies of Simulation Model Management, Software Engineering, Artificial Intelligence, and General System Theories. In: General Systems, Vol. 26, R.K. Ragade (Ed.). pp. 203–209. Also published in: Proceedings of the 26th Annual Conference of the Society for General Systems Research, Vol. 1, pp. 197–204.

    Google Scholar 

  • Ören, T.I. (1983). Quality Assurance of System Design and Model Management for Complex Problems. In: Adequate Modeling of Systems, H. Wedde (Ed.). Springer-Verlag, Heidelberg, W. Germany, pp. 205–219.

    Chapter  Google Scholar 

  • Ören T.I. 1984a. Directions for the Future of Modelling and Simulation: A Normative View. International Journal of Systems Analysis, Modelling, Simulation, Vol. 1, No. 4, pp. 293–306.

    Google Scholar 

  • Ören, T.I. (1984b — In Press). Model-Based Information Technology: Computer and System Theoretic Foundation. Behavioral Science, July 1984.

    Google Scholar 

  • Ören, T.I. (1984c — In Press). Simulation Methodology: A Top-Down View. In: Encyclopedia of Systems and Control, M. Singh (Ed.). Pergamon Press, Oxford, England.

    Google Scholar 

  • Ören, T.I. (1984d — In Press). Simulation: A Taxonomy. In: Encyclopedia of Systems and Control, M. Singh (Ed.). Pergamon Press, Oxford, England.

    Google Scholar 

  • Ören, T.I. (1984e — In Press). Simulation Models: A Taxonomy. In: Encyclopedia of Systems and Control, M. Singh (Ed.). Pergamon Press, Oxford, England.

    Google Scholar 

  • Ören, T.I. (I984, — In Press). Model Behavior: A Taxonomy of Types and Generation and Processing Techniques. In: Encyclopedia of Systems and Control, M. Singh (EH.). Pergamon Press, Oxford, England.

    Google Scholar 

  • Pugh, R.E. (1977). Evaluation of Policy Simulation Models. Information Resources Press, Washington, DC.

    Google Scholar 

  • Roth, P.F., Gass, S.I., Lemoine, A.J. (1978). Some Considerations for Improving Federal Modelling. In: Proceedings of 1978 Winter Simulation Conference, pp. 213–217.

    Google Scholar 

  • Sargent, R.G. (1978). Validation of Discrete-Event Simulation Models. In: Proceedings of NSF Workshop on Validation of Mathematical Models in Energy Related Research and Development, Forth Worth, TX.

    Google Scholar 

  • Sargent, R.G. (1982). Verification and Validation of Simulation Models. In: Progress in Modelling and Simulation, F.E. Cellier (Ed.). Academic Press London, England.

    Google Scholar 

  • Schellenberger, R.E. (1974). Criteria for Assessing Model Validity for Managerial Purposes. Decision Sciences, Vol. 5, No. 5.

    Google Scholar 

  • Schlesinger, S. et al. (1979). Terminology for Model Credibility. Simulation, Vol. 32, No. 3, pp. 103–104.

    Article  Google Scholar 

  • Schruben, L.W. (1980). Establishing the Credibility of Simulations. Simulation, Vol. 34, No. 3, pp. 101–105.

    Article  Google Scholar 

  • Theorey, T.J. (1975). Validation Criteria for Computer System Simulation. Simuletter, Vol. 6, No. 4, pp. 9–20.

    Google Scholar 

  • Thissen, W. 1978a. Investigations into the Club of Rome’s Wold 3 Model — Lessons for Understanding Complicated Models. Ph.D. Thesis, Dept. of Technical Physics, Eindhoven University of Technology, The Netherlands.

    Google Scholar 

  • Thissen, W. 1978b. Investigations into the World 3 Model: Lessons for Understanding Complicated Models. IEEE Trans, on Systems, Man, and Cybernetyics, SMC-8, No. 3, pp. 183–193.

    Google Scholar 

  • Van Horn, R. (1969). Validation. In: The Decision of Computer Experiments, T.H. Naylor (Ed.). Duke University Press, Durham, NC.

    Google Scholar 

  • Van Horn, R. (1971). Validation of Simulation Results. Management Science, Vol. 17, No. 5.

    Google Scholar 

  • Watt, K.F. (1977). Why Won’t Anyone Believe Us? Simulation, Vol. 28, No. 1, pp. 1–3.

    Article  MathSciNet  Google Scholar 

  • Wierzbicki, A. (1977). Models and Sensitivity of Control Systems. (Tn Polish). Wydawn-ictwa Naukowo-Techniczne, Warsaw, Poland

    Google Scholar 

  • Zeigler, B.P. (1976). Theory of Modelling and Simulation. Wiley-Interscience, New York.

    MATH  Google Scholar 

  • Zucker, P.A. (1983). Some Pitfalls in the Validation of Multidimensional Models. In: Proceedings of Summer Computer Simulation Conference, Vancouver, BC, Canada, pp. 187–189.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, University of Ottawa, KIN 9B4, Ottawa, Ontario, Canada

    Tuncer I. Ören

Authors
  1. Tuncer I. Ören
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Computer Science Department, University of Ottawa, Ottawa, Ontario, Canada

    Tuncer I. Ören

  2. Computer Science Department, Wayne State University, Detroit, Illinois, USA

    Bernard P. Zeigler

  3. Computer Science Department, Dutch Agricultural University, Wageningen, The Netherlands

    Maurice S. Elzas

Rights and permissions

Reprints and permissions

Copyright information

© 1984 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Ören, T.I. (1984). Quality Assurance in Modelling and Simulation: A Taxonomy. In: Ören, T.I., Zeigler, B.P., Elzas, M.S. (eds) Simulation and Model-Based Methodologies: An Integrative View. NATO ASI Series, vol 10. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-82144-8_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-82144-8_17

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-82146-2

  • Online ISBN: 978-3-642-82144-8

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation