Abstract
There is a disparity between the multitude of apparently successful expert system prototypes and the scarcity of expert systems in real everyday use. Modern tools make it deceptively easy to make reasonable prototypes, but these prototypes are seldom made subject to serious evaluation. Instead the development team confronts their product with a set of cases, and the primary evaluation criterion is the percentage of correct answers: we are faced with a “95% syndrome”. Other aspects related to the use of the system are almost ignored. There is still a long way to go from a promising prototype to a final system.
It is maintained in the article that a useful test must be performed by future users in a situation that is as realistic as possible. If this is not done claims of usefulness cannot be justified. It is also stated that prototyping does not make “traditional” analysis and design obsolete, although the contents of these activities will change.
In order to discuss the effects of using the systems a distinction between expert systems as media, tools and experts is proposed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aase, O. and Jonsbu, J. (1986). Utvikling av et edb-basert diagnosehjelpprogram for pasienter som innlegges for akutte brystsmerter. (The development of a computer-based help-program for diagnosis of patients being admitted with acute chest pain.) Technical report, Sentralsykehuset i Akershus, N-1474 Nordbyhagen.
Aikins, J. S., Kunz, J. C., Shortliffe, E. H. and Falat, R. J. (1983). PUFF: An expert system for interpretation of pulmonary function data.Computers and Biomedical Research, 16, 199–208.
Andersen, N. E., Kensing, F., Lundin, J., Mathiassen, L., Munk-Madsen, A., Rasbech, M. and Sørgaard, P. (1990).Professional Systems Development. Prentice-Hall, Business Information Technology Series, UK.
Attewell, P. and Rule, J. (1984). Computing and organizations: What we know and what we don't know.Communications of the ACM, 27(2): 1184–1192, December.
Buchanan, B. G. (1986). Expert systems: Working systems and the research literature.Expert Systems, 3(1), 32–51, January.
Buchanan, B. G.et al. (1983). Constructing an expert system. In Hayes-Roth, F., Waterman, D. A. and Lenat, D. B. (eds)Building Expert Systems, chapter 5. Addison-Wesley.
Canning, R. G. (1972). That maintenance “iceberg”.EDP Analyzer, 10(10), October.
Davis, R. (1984). Amplifying expertise with expert systems. In Winston, P. H. and Prendergast, K. (eds)The AI Business, Commercial Uses of Artifical Intelligence. MIT Press, Cambridge, Massachusetts and London.
Dreyfus, H. L. and Dreyfus, S. E. (1986).Mind over Machine, The Power of Human Intuition and Expertise in the Era of the Computer. The Free Press, New York.
Ernst, M. L. and Ojha, H. E. (1986). Business applications of artificial-intelligence KBSs.Future Generations Computer Systems, 2, 173–186.
Floyd, C. (1984). A systematic look at protoyping. In Budde, R., Kuhlenkamp, K., Mathiassen, L. and Züllighoven, H. (eds)Approaches to Prototyping, Springer-Verlag, Berlin-Heidelberg.
Göranzon, B. (1983). Bakgrunden. (The background (of the discussion).) In Göranzon, B. (ed.)Datautvecklingens filosofi (Philosophy of the development of computing). Carlsson & Jönsson, Stockholm.
Hirschheim, R. A. (1986). The effect of a priori views on the social implications of computing: The case of office automation.ACM Computing Surveys, 18(2), 165–195, June.
Lanzara, G. F. (1983). Frames, metaphors, and games. In Briefs, U., Ciborra, C. and Schneider, L. (eds)Systems Design for, with, and by the Users. North-Holland, Amsterdam.
Liebowitz, J. (1986). Useful approach for evaluating expert systems.expert Systems, 3(2).
van Melle, W. (1981).System Aids in Constructing Consultation Programs. UMI Research Press, Ann Arbor, Michigan.
Reshagen, L., Shahsavar, N., Hildén, J.-O., Hägglund, S. and Sandahl, K. (1987). Medical expert system software in MUMPS production environment. The EXODUS project. In Duisterhout, J. S. and Kirsten, W. (eds)Proceedings of the Twelfth Annual Meeting of the MUMPS Users' Group-Europe, pages 93–98, Rotterdam. MUMPS Users' Group-Europe.
van de Riet, R. P. (1987). Problems with expert systems?Future Generations Computer Systems, 3(1), 11–16, February.
Sandahl, K. (1987). Case studies in knowledge acquisition, migration and user acceptance of expert systems. Linköping Studies in Science and Technology Thesis No 127, Linköping University.
Sandahl, K., Hägglund, S., Hildén, J.-O., Rehmnert, R. and Reshagen, L. (1985). The antibody analysis advisor and its migration into a production environment. InProceedings 1st International Expert Systems Conference and Exhibition, pages 297–310, London.
Sandewall, E., Hägglund, S., Gustafsson, C., Jonesjö, L. and Strömfors, O. (1983). Stepwise structuring: a style of life for flexible software. InProceedings of the 1983 National Computer Conference, Anaheim.
Senker, P., Townsend, J. and Buckingham, J. (1989). Working with expert systems: Three case studies.AI & Society, 3(2), 103–116, April–June.
Shortliffe, E. H. (1976).Computer-based Medical Consultation: MYCIN. American Elsevier, New York.
Sinding-Larsen, H. (1987). Information technology and the management of knowledge.AI & Society, 1, 93–101.
Whitaker, R. and Östberg, O. (1988). Channeling knowledge: Expert systems as communications media.AI & Society, 2(3), 197–208, July–September.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Sørgaard, P. Evaluating expert system prototypes. AI & Soc 5, 3–17 (1991). https://doi.org/10.1007/BF01891354
Issue Date:
DOI: https://doi.org/10.1007/BF01891354