On representation of source reliability in weight of evidence
Developers of artificial intelligence-based systems have made frequent use of likelihood ratios. Those ratios have been used to represent the uncertainty associated with events and hypotheses on rules in expert systems and they have been used to establish rankings of resulting diagnoses in other systems. This paper discusses the representation of source reliability through those likelihood ratios, for use in artificial intelligence systems, such as expert systems, influence diagrams and other systems that employ a Bayesian-based approach to the representation of uncertainty to assess the weight of evidence.
This paper presents a means by which that reliability can be captured using a likelihood ratio format. Reliability can have a substantial impact on the value of the likelihood ratio. One example presented in the paper results in about a 60% decrease in the value of the likelihood ratio, with only a 10% decrease in reliability. Then this paper investigates the impact of accounting for reliability in the likelihood ratios. A monotonic property is established for the reliability embedded likelihood ratios. That property provides insight both into the behavior of weights on rules in systems that employ such an approach and into the use of likelihood ratios to rank diagnoses.
Then the impact of reliability-adjusted likelihood ratios is examined for their impact on rank ordering of the ratio. It is found that in some cases where likelihood ratios are examined in terms of the same evidence that reliability does not change the rankings. However, if the likelihood ratios are developed for comparison across different evidence, then the rankings do not remain the same. As a result, accounting for reliability of evidence can be critical to the ultimate success of systems employing this approach.
KeywordsUncertainty Representation Influence Diagrams Expert Systems Artificial Intelligence Source Reliability
Unable to display preview. Download preview PDF.
- Cooper, G. (1986) A Diagnostic method that uses causal knowledge and linear programming in the application of Bayes' formula. In Computer Methods and Programs in Biomedicine, 22 (1986), 223–237.Google Scholar
- Good, I. (1960) Weight of evidence, corroboration, explanatory power, information and the utility of experiments. In Journal of the Royal Statistical Society, B, 22 (1960), 319–331.Google Scholar
- Heckerman, D., Horvitz, E., Nathwani, B. (1990) Toward normative expert systems: The pathfinder project. Unpublished paper, University of Southern California School of Medicine and Stanford University School of Medicine, Draft.Google Scholar
- O'Leary, D. (1990) On the representation of source reliability through weights on rules. In IPMU Conference Proceedings, Paris.Google Scholar
- Schum, D., and DuCharme, W. Comments on the relationship between the impact and the reliability of evidence. In Organizational Behavior and Human Performance, 6, 111–131.Google Scholar
- Shortliffe, E. and Buchanan, B. (1985) A model of inexact reasoning in medicine," in rule-based expert systems. Addison-Wesley (B. Buchanan and E. Shortliffe, eds.).Google Scholar
- Simon, H. The sciences of the artificial, Cambridge, MA, MIT Press.Google Scholar
- Spielgelhalter, D. and Knill-Jones, R. (1984) Statistical knowledge-based approaches to clinical decision support systems, with an application in gastroenterology. In Journal of the Royal Statistical Society, 147 (1984), 35–77.Google Scholar