Advertisement

Empirical Evaluation of User Models and User-Adapted Systems

  • David N. Chin
Article

Abstract

Empirical evaluations are needed to determine which users are helped or hindered by user-adapted interaction in user modeling systems. A review of past UMUAI articles reveals insufficient empirical evaluations, but an encouraging upward trend. Rules of thumb for experimental design, useful tests for covariates, and common threats to experimental validity are presented. Reporting standards including effect size and power are proposed.

empirical evaluation experimental design covariant variables effect size treatment magnitude power sensitivity 

References

  1. Albrecht, D. W., Zukerman, I. and Nicholson, A. E.: 1998, Bayesian models for keyhole plan recognition in an adventure game. User Modeling and User-Adapted Interaction 8(1–2), 5-47.Google Scholar
  2. Alspector, J., Kołcz, A. and Karunanithi, N.: 1997, Feature-based and clique-based user models for movie selection: a comparative study. User Modeling and User-Adapted Interaction 7(4), 279-304.Google Scholar
  3. Balabanović, M.: 1998, Exploring versus exploiting when learning user models for text recommendation. User Modeling and User-Adapted Interaction 8(1–2), 71-102.Google Scholar
  4. Boyle, C. and Encarnacion, A. O.: 1994, MetaDoc: An adaptive hypertext reading system. User Modeling and User-Adapted Interaction 4(1), 1-19.Google Scholar
  5. Brusilovsky, P.: 1996, Methods and techniques of adaptive hypermedia. User Modeling and User-Adapted Interaction 6(2–3), 87-129.Google Scholar
  6. Brusilovsky, P.: 1998, Methods and techniques of adaptive hypermedia. In: P. Brusilovsky, A. Kobsa and J. Vassileva (eds.), Adaptive Hypertext and Hypermedia. Kluwer Academic Publishers, Dordrecht, pp. 1-43.Google Scholar
  7. Brusilovsky, P.: 2001, Adaptive hypermedia. User Modeling and User-Adapted Interaction 11(1–2), 87-110 (this issue).Google Scholar
  8. Brusilovsky, P. and Eklund, J.: 1998, A study of user-model based link annotation in educational hypermedia. Journal of Universal Computer Science 4(4), 429-448.Google Scholar
  9. Calistri-Yeh, R. J.: 1991, Utilizing user models to handle ambiguity and misconceptions in robust plan recognition. User Modeling and User-Adapted Interaction 1(4), 289-322.Google Scholar
  10. Campbell, D. T. and Stanley, J. C.: 1963, Experimental and quasi-experimental designs for research. In: N. L. Gage (ed.), Handbook of Research on Teaching. Rand McNally and Co., Chicago.Google Scholar
  11. Carberry, S.: 2001, Techniques for Plan Recognition. User Modeling and User-Adapted Interaction 11(1–2), 31-48 (this issue).Google Scholar
  12. Carbonaro, A., Maniezzo, V., Roccetti, M. and Salomoni, P.: 1995, Modelling the student in Pitagora 2.0. User Modeling and User-Adapted Interaction 4(4), 233-251.Google Scholar
  13. Chiu, B. C. and Webb, G. I.: 1998, Using decision trees for agent modeling: Improving prediction performance. User Modeling and User-Adapted Interaction 8(1–2), 131-152.Google Scholar
  14. Chu-Carroll, J. and Brown, M. K.: 1998, An evidential model for tracking initiative in collaborative dialogue interactions. User Modeling and User-Adapted Interaction 8(3–4), 215-253.Google Scholar
  15. Corbett, A.T., Anderson, J. R. and O'Brien, A. T.: 1993, The predictive validity of student modeling in the ACT programming tutor. In: P. Brna, S. Ohlsson and H. Pain (eds.), Artificial Intelligence and Education, 1993: The Proceedings of AI-ED 93. Charlottesville, VA: AACE.Google Scholar
  16. Corbett, A. T. and Anderson, J. R.: 1995, Knowledge tracing: modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4(4), 253-278.Google Scholar
  17. Debevc, M., Meyer, B., Donlagic, D. and Svecko, R.: 1996, Design and evaluation of an adaptive icon toolbar. User Modeling and User-Adapted Interaction 6(1), 1-21.Google Scholar
  18. Fischer, G.: 2001, User modeling in human-computer interaction. User Modeling and User-Adapted Interaction 11(1–2), 65-86 (this issue).Google Scholar
  19. Gatti, G.G. and Harwell, M.: 1998, Advantages of computer programs over power charts for the estimation of power. Journal of Statistics Education 6(3). Also available at www.amstat.org/publications/jse/v6n3/gatti.html.Google Scholar
  20. Gmytrasiewicz, P. J. and Noh, S. and Kellogg, T.: 1998, Bayesian update of recursive agent models. User Modeling and User-Adapted Interaction 8(1–2), 49-69.Google Scholar
  21. Green, N. and Carberry, S.: 1999, A computational mechanism for initiative in answer generation. User Modeling and User-Adapted Interaction 9(1–2), 93-132.Google Scholar
  22. Guinn, C. I.: 1998, An analysis of initiative selection in collaborative task-oriented discourse. User Modeling and User-Adapted Interaction 8(3–4), 255-314.Google Scholar
  23. Hirashima, T., Kashihara, A. and Toyoda, J.: 1997, Information filtering using user's context on browsing in hypertext. User Modeling and User-Adapted Interaction 7(4), 239-256.Google Scholar
  24. Huck, S., Cormier, W. and Bounds, W.: 1974, Reading Statistics and Research. Harper and Row, New York.Google Scholar
  25. Ishizaki, M., Crocker, M. and Mellish, C.: 1999, Mixed-initiative dialogue using computer dialogue simulation. User Modeling and User-Adapted Interaction 9(1–2), 79-91.Google Scholar
  26. Illeényi, A. and Korpaãssy, P.: 1981, Correlation between loudness and quality of stereophonic loudspeakers. Acustica 49(4), 334-336.Google Scholar
  27. Jennings, A. and Higuchi, H.: 1993, A user model neural network for a personal news service. User Modeling and User-Adapted Interaction 3(1), 1-25.Google Scholar
  28. Kaplan, C., Fenwick, J. and Chen, J.: 1993, Adaptive hypertext navigation based on user goals and context. User Modeling and User-Adapted Interaction 3(3), 193-220.Google Scholar
  29. Kashihara, A., Hirashima, T. and Toyoda, J.: 1995, A cognitive load application in tutoring. User Modeling and User-Adapted Interaction 4(4), 279-303.Google Scholar
  30. Keppel, G.: 1991, Design and Analysis: A Researcher's Handbook 3rd Edn. Prentice-Hall, Englewood Cliffs, NJ.Google Scholar
  31. Krause, J., Hirschmann, A. and Mittermaier, E.: 1993, The intelligent help system COM-FOHELP: Towards a solution of the practicability problem for user modeling and adaptive systems. User Modeling and User-Adapted Interaction 3(3), 249-282.Google Scholar
  32. Lipschitz, S. P. and Van der Kooy, J.: 1981, The great debate: Subjective evaluation. Journal of the Audio Engineering Society 29(7/8), 482-491.Google Scholar
  33. London, R.: 1992, Student modeling to support multiple instructional approaches. User Modeling and User-Adapted Interaction, 2(1–2), 117-154.Google Scholar
  34. Marshall, C. and Rossman, G. B.: 1999, Designing Qualitative Research 3rd Edn. Alta Mira Press, Walnut Creek, CA.Google Scholar
  35. Matheé, N. and Chen, J. R.: 1996, User-centered indexing for adaptive information access. User Modeling and User-Adapted Interaction 6(2–3), 225-261.Google Scholar
  36. Miles, M. B. and Huberman, A. H.: 1994, Qualitative Data Analysis: An Expanded Sourcebook 2nd Edn. Sage Publications, London.Google Scholar
  37. Milne, S., Shiu, E. and Cook, J.: 1996, Development of a model of user attributes and its implementation within an adaptive tutoring system. User Modeling and User-Adapted Interaction 6(4), 303-335.Google Scholar
  38. Mitrovic, A., Djordjevic-Kajan, S. and Stoimenov, L.: 1996, INSTRUCT: Modeling students by asking questions. User Modeling and User-Adapted Interaction 6(4), 273-302.Google Scholar
  39. Neter, J., Wasserman, W., and Kutner, M. H.: 1985, Applied Linear Statistical Models 2nd Edn. Richard D. Irvin, Homewood, IL.Google Scholar
  40. Newell, S.: 1997, User models and ¢ltering agents for improved internet information retrieval. User Modeling and User-Adapted Interaction 7(4), 223-237.Google Scholar
  41. Nwana, H. S.: 1991, User modelling and user adapted interaction in an intelligent tutoring system. User Modeling and User-Adapted Interaction 1(1), 1-32.Google Scholar
  42. Raskutti, B., Beitz, A. and Ward, B.: 1997, A feature-based approach to recommending selections based on past preferences. User Modeling and User-Adapted Interaction 7(3), 179-218.Google Scholar
  43. Shanefield, D.: 1980, The great ego crunchers: Equalized, double-blind tests. High Fidelity, March, 57-61.Google Scholar
  44. Shute, V. J.: 1995, SMART: Student modeling approach for responsive tutoring. User Modeling and User-Adapted Interaction 5(1), 1-44.Google Scholar
  45. Silverman, D.: 1993, Interpreting qualitative data: Methods for analysing talk, text and interaction. Sage Publications, Thousand Oaks, CA.Google Scholar
  46. Sison, R. C., Numao, M. and Shimura, M.: 1998, Discovering error classes from discrepancies in novice behaviors via multistrategy conceptual clustering. User Modeling and User-Adapted Interaction 8(1–2), 103-129.Google Scholar
  47. Specht, M. and Kobsa, A.: 1999, Interaction of domain expertise and interface design in adaptive educational hypermedia. In Workshops on Adaptive Systems and User Modeling on the World Wide Web at WWW-8, Toronto, Canada and UM99, Banff, Canada.Google Scholar
  48. Stevens, J.: 1992, Applied Multivariate Statistics for the Social Sciences 2nd Edn. Lawrence Erlbaum, Hillsdale, NJ.Google Scholar
  49. Tattersall, C.: 1992, Generating help for users of application software. User Modeling and User-Adapted Interaction, 2(3), 211-248.Google Scholar
  50. Vassileva, J.: 1996, A task-centered approach for user modeling in a hypermedia office documentation system. User Modeling and User-Adapted Interaction 6(2–3), 185-224.Google Scholar
  51. Virvou, M. and du Bulay, B.: 1999, Human plausible reasoning for intelligent help. User Modeling and User-Adapted Interaction 9(4), 323-377.Google Scholar
  52. Webb, G. I. and Kuzmycz, M.: 1996, Feature based modelling: a methodology for producing coherent, consistent, dynamically changing models of agent's competencies. User Modeling and User-Adapted Interaction 5(2), 117-150.Google Scholar
  53. Webb, G. I., Pazzani, M. J. and Billsus, D.: 2000, Machine learning for user modeling. User Modeling and User-Adapted Interaction 11(1–2), 19-29 (this issue).Google Scholar
  54. Weber, R. P.: 1990, Basic Content Analysis 2nd Edn. Sage Publications, Beverly Hills.Google Scholar
  55. Yin, R. K.: 1994, Case Study Research: Design and Methods 2nd Edn. Sage Publications, London.Google Scholar
  56. Zukerman, I. and Albrecht, D. W.: 2000, Predictive statistical user models for user modeling. User Modeling and User-Adapted Interaction 11(1–2), 5-18 (this issue).Google Scholar
  57. Zukerman, I. and Litman, D.: 2000, Natural language processing and user modeling: Synergies and limitations. UserModeling and User-Adapted Interaction 11(1–2), 129-158 (this issue).Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • David N. Chin
    • 1
  1. 1.Department of Information and Computer SciencesUniversity of HawaiiHonoluluU.S.A.

Personalised recommendations