Evaluation of Incremental Knowledge Acquisition with Simulated Experts

  • Paul Compton
  • Tri M. Cao
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4304)


Evaluation of knowledge acquisition (KA) is difficult in general. In recent times, incremental knowledge acquisition that emphasises direct communication between human experts and systems has been increasingly widely used. However, evaluating incremental KA techniques, like KA in general, has been difficult because of the costs of using human expertise in experimental studies. In this paper, we use a general simulation framework to evaluate Ripple Down Rules (RDR), a successful incremental KA method. We focus on two fundamental aspects of incremental KA: the importance of acquiring domain ontological structures and the usage of cornerstone cases.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beydoun, G., Hoffmann, A.: Incremental acquisition of search knowledge. Journal of Human-Computer Studies 52, 493–530 (2000)CrossRefGoogle Scholar
  2. 2.
    Cao, T., Compton, P.: A simulation framework for knowledge acquisition evaluation. In: Proceedings of 28th Australasian Computer Science Conference, pp. 353–361 (2005)Google Scholar
  3. 3.
    Clancey, W.J.: Heuristic classification. Artificial Intelligence 27, 289–350 (1985)CrossRefGoogle Scholar
  4. 4.
    Compton, P., Edwards, G., Srinivasan, A., Malor, P., Preston, P., Kang, B., Lazarus, L.: Ripple-down-rules: Turning knowledge acquisition into knowledge maintenance. Artificial Intelligence in Medicine 4, 47–59 (1992)CrossRefGoogle Scholar
  5. 5.
    Compton, P., Jansen, R.: A philosophical basis for knowledge acquisition. Knowledge Acquisition 2, 241–257 (1990)CrossRefGoogle Scholar
  6. 6.
    Compton, P., Preston, P., Kang, B.: The use of simulated experts in evaluating knowledge acquisition. In: Gaines, B., Musen, M. (eds.) 9th Banff KAW Proceeding, pp. 1–12 (1995)Google Scholar
  7. 7.
    Compton, P.: Simulating expertise. In: PKAW, pp. 51–70 (2000)Google Scholar
  8. 8.
    Compton, P., Peters, L., Edwards, G., Lavers, T.: Experience with ripple-down rules. In: Applications and Innovations in Intelligent Systems, pp. 109–121 (2005)Google Scholar
  9. 9.
    Gomez-Perez, A.: Evaluation of ontologies. Int. J. Intelligent Systems 16, 391–409 (2001)zbMATHCrossRefGoogle Scholar
  10. 10.
    Kang, B., Yoshida, K., Motoda, H., Compton, P.: A help desk system with intelligence interface. Applied Artificial Intelligence 11, 611–631 (1997)CrossRefGoogle Scholar
  11. 11.
    Menzies, T., Van Hamelen, F.: Editorial: Evaluating knowledge engineering techniques. Journal of Human-Computer Studies 51(4), 715–727 (1999)CrossRefGoogle Scholar
  12. 12.
    Noy, N.F., Sintek, M., Decker, S., Crubézy, M., Fergerson, R.W., Musen, M.A.: Creating semantic web contents with protégé-2000. IEEE Intelligent Systems 16(2), 60–71 (2001)CrossRefGoogle Scholar
  13. 13.
    Preston, P., Edwards, G., Compton, P.: A 2000 rule expert system without a knowledge engineer. In: Gaines, B., Musen, M. (eds.) 8th Banff KAW Proceeding (1994)Google Scholar
  14. 14.
    Schreiber, G., Wielinga, B.J., de Hoog, R., Akkermans, H., Van de Velde, W.: Commonkads: A comprehensive methodology for kbs development. IEEE Expert 9(6), 28–37 (1994)CrossRefGoogle Scholar
  15. 15.
    Shadbolt, N., O’Hara, K.: The experimental evaluation of knowledge acquisition techniques and methods: history, problem and new directions. Journal of Human-Computer Studies 51(4), 729–775 (1999)CrossRefGoogle Scholar
  16. 16.
    Shiraz, G., Sammut, C.: Combining knowledge acquisition and machine learning to control dynamic systems. In: Proceedings of the 15th International Joint Conference in Artificial Intelligence (IJCAI 1997), pp. 908–913. Morgan Kaufmann, San Francisco (1997)Google Scholar
  17. 17.
    Sure, Y., Gómez-Pérez, A., Daelemans, W., Reinberger, M.-L., Guarino, N., Noy, N.F.: Why evaluate ontology technologies? because it works! IEEE Intelligent Systems 19(4), 74–81 (2004)CrossRefGoogle Scholar
  18. 18.
    van Heijst, G., Schreiber, A.Th., Wielinga, B.J.: Using explicit ontologies in kbs development. Journal of Human-Computer Studies 45, 183–292 (1997)Google Scholar
  19. 19.
    van Heijst, G., Terpstra, P., Wielinga, B.J., Shadbolt, N.: Using generalised directive models in knowledge acquisition. In: Wetter, T., Boose, J., Linster, M., Althoff, K.-D., Gaines, B.R., Schmalhofer, F. (eds.) EKAW 1992. LNCS, vol. 599, pp. 112–132. Springer, Heidelberg (1992)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Paul Compton
    • 1
  • Tri M. Cao
    • 1
  1. 1.School of Computer Science and EngineeringUniversity of New South WalesSydneyAustralia

Personalised recommendations