Using Decision Trees for Agent Modeling: Improving Prediction Performance

  • Bark Cheung Chiu
  • Geoffrey I. Webb
Article

Abstract

A modeling system may be required to predict an agent's future actions under constraints of inadequate or contradictory relevant historical evidence. This can result in low prediction accuracy, or otherwise, low prediction rates, leaving a set of cases for which no predictions are made. A previous study that explored techniques for improving prediction rates in the context of modeling students' subtraction skills using Feature Based Modeling showed a tradeoff between prediction rate and predication accuracy. This paper presents research that aims to improve prediction rates without affecting prediction accuracy. The FBM-C4.5 agent modeling system was used in this research. However, the techniques explored are applicable to any Feature Based Modeling system, and the most effective technique developed is applicable to most agent modeling systems. The default FBM-C4.5 system models agents' competencies with a set of decision trees, trained on all historical data. Each tree predicts one particular aspect of the agent's action. Predictions from multiple trees are compared for consensus. FBM-C4.5 makes no prediction when predictions from different trees contradict one another. This strategy trades off reduced prediction rates for increased accuracy. To make predictions in the absence of consensus, three techniques have been evaluated. They include using voting, using a tree quality measure and using a leaf quality measure. An alternative technique that merges multiple decision trees into a single tree provides an advantage of producing models that are more comprehensible. However, all of these techniques demonstrated the previous encountered trade-off between rate of prediction and accuracy of prediction, albeit less pronounced. It was hypothesized that models built on more current observations would outperform models built on earlier observations. Experimental results support this hypothesis. A Dual-model system, which takes this temporal factor into account, has been evaluated. This fifth approach achieved a significant improvement in prediction rate without significantly affecting prediction accuracy.

Agent modeling Student modeling Inductive learning Decision tree. 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ali, K., Brunk, C., and Pazzani, M.: 1994, On learning multiple descriptions of a concept. In Proceedings of the Sixth International Conference on Tools with Artificial Intelligence, New Orleans, LA: IEEE Press, 476-483.Google Scholar
  2. Amato, N. H. and Tsang, C. P.: 1990, Student modeling in a keyboard scale tutoring system. In C. J. Barter & M. J. Brooks (Eds.) Proceedings of the Second Australian Joint Conference on Artificial Intelligence, Berlin: Springer-Verlag, 225-239.Google Scholar
  3. Anderson, J. R., Boyle, C. F., and Reiser, B. J.: 1985, Intelligent tutoring systems. Science, 228 456-462.Google Scholar
  4. Anderson, J. R., Boyle, C. F., Corbett, A. T., and Lewis, M. W.: 1990, Cognitive modeling and intelligent tutoring. Artificial Intelligence, 42 7-49.Google Scholar
  5. Baffes, P., and Mooney, R.: 1996, Refinement-based student modeling and automated bug library construction. Journal of Artificial Intelligence in Education, 7(1) 75-117.Google Scholar
  6. Balabanovic, M.: 1998, Exploring versus exploiting when learning user models for text recommendation. In this volume.Google Scholar
  7. Breiman, L.: 1996, Bagging predictors. Machine Learning, 24 123-140.Google Scholar
  8. Brown, J. S., and VanLehn, K.: 1980, Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4 379-426.Google Scholar
  9. Brown, J. S., and Burton, R. R.: 1978, Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 2 155-192.Google Scholar
  10. Burton, R. R, and Brown, J. S.: 1976, A tutoring and student modeling paradigm for gaming environments. Computer Science and Education. ACM SIGCSE Bulletin, 8(1) 236-246.Google Scholar
  11. Carbonell, J. R.: 1970, AI in CAI: An artificial intelligence approach to computer-assisted instruction. IEEE Transactions on Man-Machine Systems, 11(4) 190-202.Google Scholar
  12. Chan, P. K., and Stolfo, S. J.: 1995, A comparative evaluation of voting and meta-learning on Partitioned data. In Prieditis, A., and Russell, S., eds., Proceedings of the 12th International Conference on Machine Learning, 90-98.Google Scholar
  13. Chiu, B. C., Webb, G. I., and Kuzmycz, M.: 1997, A comparison of first-order and zeroth-order induction for input-output agent modeling. In Jameson A., Paris C., and Tasso C., eds., Proceedings of the Sixth International Conference on User Modeling, UM97, 347-358.Google Scholar
  14. Corbett, A. T., and Anderson, J. R.: 1992, Student modeling and mastery learning in a computer-based programming tutor. In Frasson, C., Gauthier, G., and McCalla, G. I., eds., Intelligent Tutoring Systems. Berlin: Springer-Verlag, 413-420.Google Scholar
  15. Desmoulins, C., and Van Labeke, N.: 1996, Towards student modeling in geometry with inductive logic programming. In Brna, P., Paiva, A., and Self, J., eds., Proceedings of the European Conference on Artificial Intelligence in Education, 94-100.Google Scholar
  16. Dietterich, T., and Bakiri, G.: 1994, Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2263-286.Google Scholar
  17. Giangrandi, P., and Tasso, C.:1995, Truth maintenance techniques for modeling students' behavior. Journal of Artificial Intelligence in Education, 6(2/3) 153-202.Google Scholar
  18. Giangrandi, P., and Tasso, C.: 1996, Modeling the temporal evolution of student's knowledge. In Brna, P., Paiva, A., and Self, J., eds., Proceedings of the European Conference on Artificial Intelligence in Education, 184-190.Google Scholar
  19. Gilmore, D., and Self, J.: 1988, The application of machine learning to intelligent tutoring systems. In Self, J., ed., Artificial Intelligence and Human Learning: Intelligent Computer-aided Instruction. London: Chapman and Hall, 179-196.Google Scholar
  20. Goldstein, I. P.: 1979, The genetic graph: A representation for the evolution of procedural knowledge. International Journal of Man-machine Studies, 11(1) 51-77.Google Scholar
  21. Hoppe, H. U.: 1994, Deductive error diagnosis and inductive error generalization for intelligent tutoring systems. Journal of Artificial Intelligence in Education, 5(1) 27-49.Google Scholar
  22. Heath D., Kasif, S. and Salzberg, S.: 1996, Committees of decision trees. In Gorayska, B., and Mey, J., eds., Cognitive Technology: In Search of a Human Interface. Amsterdam: Elsevier Science B.V., 305-317.Google Scholar
  23. Ikeda, M., Kono, Y., and Mizoguchi, R.: 1993, Nonmonotonic model inference: A formalization of student modeling. Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence: IJCAI'93, 467-473.Google Scholar
  24. Kohavi, R.: 1995, A study of cross-validation and bootstrap for accuracy estimation and model selection. In Mellish, C. S., ed., Proceedings of the 14th International Joint Conference on Artificial Intelligence. Morgan Kaufmann, 1137-1145.Google Scholar
  25. Kuzmycz, M.: 1994, A dynamic vocabulary for student modeling. Proceedings of the Fourth International Conference on User Modeling, Hyannis, MA, 185-190.Google Scholar
  26. Kuzmycz, M.: 1997, Resolving conflicting knowledge in student models. Proceedings of the Eighth World Conference on Artificial Intelligence in Education. Amsterdam: IOS Press, 522-529.Google Scholar
  27. Kuzmycz, M., and Webb, G. I.: 1992, Evaluation of feature based modelling in subtraction. In Frasson, C., Gauthier, G., and McCalla, G. I., eds., Intelligent Tutoring Systems. Berlin: Springer-Verlag, 269-276.Google Scholar
  28. Kwok, S., and Carter, C.: 1990, Multiple decision trees. Uncertainty in Artificial Intelligence, 4 327-335.Google Scholar
  29. Langley, P., and Ohlsson, S.: 1984, Automated cognitive modeling. Proceedings of the National Conference on Artificial Intelligence, Austin, Texas, 193-197.Google Scholar
  30. Langley, P., Wogulis, J., and Ohlsson, S.: 1990, Rules and principles in cognitive diagnosis. Diagnostic Monitoring of Skill and Knowledge Acquisition. Hillsdale, NJ: Erlbaum, 217-250.Google Scholar
  31. Martin, J., and VanLehn, K.: 1995, Student assessment using Bayesian nets. International Journal of Human-Computer Studies, 42(6) 575-591.Google Scholar
  32. Nock, R., and Cascuel, O.: 1995, On learning decision committees. Proceedings of the Twelfth International Conference on Machine Learning, San Mateo, CA: Morgan Kaufmann. 413-420.Google Scholar
  33. Ohlsson, S., and Langley, P.: 1985, Identifying solution paths in cognitive diagnosis. Technical Report CMU-RI-TR-85-2, Carnegie-Mellon University, Pittsburgh, PA.Google Scholar
  34. Oliver, J. J., and Hand, D. J.: 1995, On pruning and averaging decision trees. Proceedings of the Twelfth International Conference on Machine Learning, San Mateo, CA: Morgan Kaufmann, 430-437.Google Scholar
  35. Quinlan, J. R.: 1986, The effect of noise on concept learning. In Michalski, R. S., Carbonell. G. and Mitchell T.M. eds., Machine Learning: An Artificial Intelligence Approach, Vol. 2. San Mateo, CA: Morgan Kaufmann, 149-166.Google Scholar
  36. Quinlan, J. R.: 1993, C4.5: Programs for Machine Learning San Mateo, CA: Morgan Kaufmann.Google Scholar
  37. Salvia, A. A.: 1990, Introduction to Statistics. Saunders College Pub., Philadelphia.Google Scholar
  38. Schapire, R. E.: 1990, The strength of weak learnability. Machine Learning, 5 197-227.Google Scholar
  39. Sleeman, D.: 1982, Assessing aspects of competence in basic algebra. In Sleeman, D. H., and Brown, J. S., eds., Intelligent Tutoring Systems. London: Academic Press, 185-199.Google Scholar
  40. Sleeman, D., Ward, R. D., Kelly, E., Martinak, R., and Moore, J.: 1991, An overview of recent studies with Pixie. In Goodyear, P., ed., Teaching Knowledge and Intelligent Tutoring. Norwood, NJ: Ablex, 173-185.Google Scholar
  41. Ting, K. M., and Low, B. T.: 1997, Model combination in the multiple-data-batches scenario. Proceedings of the Ninth European Conference on Machine Learning, Lecture Notes in Artificial Intelligence, Vol. 1224, Berlin: Springer-Verlag, 250-265.Google Scholar
  42. Webb, G. I.: 1989, A machine learning approach to student modeling. Proceedings of the Third Australian Joint Conference on Artificial Intelligence, Melbourne, 195-205.Google Scholar
  43. Webb, G. I.: 1991, Inside the Unification Tutor: The architecture of an intelligent educational system. In Proceedings of the Fourth Australian Society for Computers in Learning in Tertiary Education Conference, Launceston, 677-684.Google Scholar
  44. Webb, G. I., Chiu, B. C., and Kuzmycz, M.: 1997, Comparative evaluation of alternative induction engines for Feature Based Modelling. International Journal of Artificial Intelligence in Education, 8, to be printed.Google Scholar
  45. Webb, G. I., and Kuzmycz, M.: 1996, Feature based modeling: A methodology for producing coherent, dynamically changing models of agent's competencies. User Modeling and User-Adapted Interaction, 5(2) 117-150.Google Scholar
  46. Wolpert, D. H.: 1992, Stacked generalisation. Neural Networks, 5 241-259.Google Scholar
  47. Young, R., and O'Shea, T.: 1981, Errors in children's subtraction. Cognitive Science, 5(1) 153-177.Google Scholar
  48. Zheng, Z., and Webb, G. I.: 1997, Lazy Bayesian Tree. Technical Report TC97/07, School of computing and mathematics, Deakin University.Google Scholar

Copyright information

© Kluwer Academic Publishers 1998

Authors and Affiliations

  • Bark Cheung Chiu
    • 1
  • Geoffrey I. Webb
    • 1
  1. 1.School of Computing and MathematicsDeakin UniversityAustralia

Personalised recommendations