Usability Engineering for the Adaptive Web

  • Cristina Gena
  • Stephan Weibelzahl
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4321)

Abstract

This chapter discusses a usability engineering approach for the design and the evaluation of adaptive web-based systems, focusing on practical issues. A list of methods will be presented, considering a user-centered approach. After having introduced the peculiarities that characterize the evaluation of adaptive web-based systems, the chapter describes the evaluation methodologies following the temporal phases of evaluation, according to a user-centered approach. Three phases are distinguished: requirement phase, preliminary evaluation phase, and final evaluation phase. Moreover, every technique is classified according to a set of parameters that highlight the practical exploitation of that technique. For every phase, the appropriate techniques are described by giving practical examples of their application in the adaptive web. A number of issues that arise when evaluating an adaptive system are described, and potential solutions and workarounds are sketched.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Alepis, E., Virvou, M.: User Modelling: An Empirical Study for Affect Perception Through Keyboard and Speech in a Bi-modal User Interface. In: Wade, V., Ashman, H., Smyth, B. (eds.) AH 2006. LNCS, vol. 4018, pp. 338–341. Springer, Heidelberg (2006)Google Scholar
  2. 2.
    Alfonseca, E., Rodriguezi, P.: Modelling Users’ Interests and Needs for an Adaptive On-line Information System. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 76–82. Springer, Heidelberg (2003)Google Scholar
  3. 3.
    Altman, D., Gore, S., Gardner, M., Pocock, S.: Statistical Guidelines for Contributors to Medical Journals. British Medical Journal 286, 1489–1493 (1983)Google Scholar
  4. 4.
    Anderson, J.R., Boyle, C.F., Yost, G.: The Geometry Tutor. In: 9th International Joint Confenrence on AI, pp. 1–7 (1985)Google Scholar
  5. 5.
    Ardissono, L., Gena, C., Torasso, P., Bellifemine, F., Chiarotto, A., Difino, A., Negro, B.: User Modeling and Recommendation Techniques for Personalized Electronic Program Guides. In: Ardissono, L., Kobsa, A., Maybury, M. (eds.) Personalization and user-adaptive interaction in digital tv, pp. 30–36. Kluwer Academic Publishers, Dordrecht (2004)Google Scholar
  6. 6.
    Bakeman, R., Gottman, J.M.: Observing Behavior: An Introduction to Sequential Analysis. Cambridge University Press, Cambridge (1986)Google Scholar
  7. 7.
    Bares, W.H., Lester, J.C.: Cinematographic User Models for Automated Realtime Camera Control in Dynamic 3D Environments. In: Jameson, A., Paris, C., Tasso, C. (eds.) User modeling: Proceedings of the Sixth International Conference, UM’97, pp. 215–226. Springer, New York (1997)Google Scholar
  8. 8.
    Barker, T., Jones, S., Britton, C., Messer, D.: The Use of a Co-operative Student Model of Learner Characteristics to Configure a Multimedia Application. User Modeling and User-adaptive Interaction 12(2), 207–241 (2002)MATHGoogle Scholar
  9. 9.
    Baudisch, P., Brueckner, L.: TV Scout: Lowering the Entry Barrier to Personalized TV Program Recommendation. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) AH 2002. LNCS, vol. 2347, pp. 58–68. Springer, Heidelberg (2002)Google Scholar
  10. 10.
    Beck, J.E., Jia, P., Sison, J., Mostow, J.: Predicting Student Help-request Behavior in an Intelligent Tutor for Reading. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 303–312. Springer, Heidelberg (2003)Google Scholar
  11. 11.
    Beck, J.E., Jia, P., Sison, J., Mostow, J.: Assessing Student Proficiency in a Reading Tutor That Listens. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 323–327. Springer, Heidelberg (2003)Google Scholar
  12. 12.
    Beck, J.E., Stern, M., Woolf, B.P.: Using the Student Model to Control Problem Difficulty. In: Jameson, A., Paris, C., Tasso, C. (eds.) User modeling: Proceedings of the Sixth International Conference, UM’97, pp. 277–288. Springer, New York (1997)Google Scholar
  13. 13.
    Bednarik, R.: Potentials of Eye-Movement Tracking in Adaptive Systems. In: Weibelzahl, S., Paramythis, A., Masthoff, J. (eds.) Proceedings of the Fourth Workshop on Empirical Evaluation of Adaptive Systems, held at the 10th International Conference on User Modeling UM2005, Edinburgh, pp. 1–8 (2005)Google Scholar
  14. 14.
    Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D., Olkin, I., Pitkin, R., Rennie, D., Schultz, K., Simel, D., Stroup, D.: Improving the Quality of Reporting Randomized Trials (the CONSORT Statement). Journal of the American Medical Association 276(8), 637–639 (1996)Google Scholar
  15. 15.
    Bellotti, V., MacLean, A.: Design Space Analysis (DSA), http://www.mrc-cbu.cam.ac.uk/amodeus/summaries/DSAsummary.html
  16. 16.
    Bental, D., Cawsey, A., Pearson, J., Jones, R.: Does Adapted Information Help Patients with Cancer? In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 288–291. Springer, Heidelberg (2003)Google Scholar
  17. 17.
    Benyon, D.: Adaptive Systems: A Solution to Usability Problems. User Modeling and User-adaptive Interaction 3, 65–87 (1993)Google Scholar
  18. 18.
    Beyer, H., Holtzblatt, K.: Contextual Design: Defining Customer-Centered Systems. Morgan Kaufmann Publishers, San Francisco (1998)Google Scholar
  19. 19.
    Billsus, D., Pazzani, M.: Learning Collaborative Information Filters. In: Proceedings of the International Conference on Machine Learning, Morgan Kaufmann, San Francisco (1998)Google Scholar
  20. 20.
    Brunstein, A., Jacqueline, W., Naumann, A., Krems, J.F.: Learning Grammar with Adaptive Hypertexts: Reading or Searching? In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) AH 2002. LNCS, vol. 2347, Springer, Heidelberg (2002)Google Scholar
  21. 21.
    Brusilovsky, P., Eklund, J.: A study of user-model based link annotation in educational hypermedia. Journal of Universal Computer Science, Special Issue on Assessment Issues for Educational Software 4(4), 429–448 (1998)Google Scholar
  22. 22.
    Brusilovsky, P., Karagiannidis, C., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: Weibelzahl, S., Chin, D.N., Weber, G. (eds.) Empirical Evaluation of Adaptive Systems. Proceedings of Workshop At the Eighth International Conference on User Modeling, UM2001, pp. 1–8 (2001)Google Scholar
  23. 23.
    Brusilovsky, P., Millán, E.: User Models for Adaptive Hypermedia and Adaptive Educational Systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization. LNCS, vol. 4321, pp. 3–53. Springer, Heidelberg (2007)Google Scholar
  24. 24.
    Brusilovsky, P., Pesin, L.: Adaptive navigation support in educational hypermedia: An evaluation of the ISIS-tutor. Journal of Computing and Information Technology 6(1), 27–38 (1998)Google Scholar
  25. 25.
    Brusilovsky, P., Sosnovsky, S., Yudelson, M.: Addictive Links: The Motivational Value of Adaptive Link Annotation in Educational Hypermedia. In: Wade, V., Ashman, H., Smyth, B. (eds.) AH 2006. LNCS, vol. 4018, pp. 51–60. Springer, Heidelberg (2006)Google Scholar
  26. 26.
    Bull, S.: User Modelling and Mobile Learning. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 383–387. Springer, Heidelberg (2003)Google Scholar
  27. 27.
    Bunt, A.: User Modeling to support user customization. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 499–501. Springer, Heidelberg (2005)Google Scholar
  28. 28.
    Burke, R.: Hybrid Recommender Systems: Survey and Experiments. User Modeling and User-Adapted Interaction 12(4), 331–370 (2002)MATHGoogle Scholar
  29. 29.
    Burke, R.: Hybrid Web Recommender Systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization. LNCS, vol. 4321, pp. 377–408. Springer, Heidelberg (2007)Google Scholar
  30. 30.
    Cena, F., Gena, C., Modeo, S.: How to communicate recommendations? Evaluation of an adaptive annotation technique. In: Costabile, M.F., Paternó, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 1030–1033. Springer, Heidelberg (2005)Google Scholar
  31. 31.
    Chapanis, A.: Evaluating usability. In: Shackel, B., Richardson, S.J. (eds.) Human Factors for Informatics Usability, pp. 359–395. Cambridge University Press, Cambridge (1991)Google Scholar
  32. 32.
    Chin, D.N.: Empirical evaluation of user models and user-adapted systems. User Modeling and User-Adapted Interaction 11(1-2), 181–194 (2001)MATHGoogle Scholar
  33. 33.
    Cohen, J.: Statistical power analysis for the behavioral sciences (revised edn.). Academic Press, New York (1977)Google Scholar
  34. 34.
    Conati, C., Merten, C., Muldner, K.: Exploring Eye Tracking to Increase Bandwidth. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 357–366. Springer, Heidelberg (2005)Google Scholar
  35. 35.
    Console, L., Gena, C., Torre, I.: Evaluation of an On-vehicle Adaptive Tourist Service. In: Weibelzahl, S., Paramythis, A. (eds.) Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh, pp. 51–60 (2003)Google Scholar
  36. 36.
    Courage, C., Baxter, K.: Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and Techniques. Morgan Kaufmann, San Francisco (2005)Google Scholar
  37. 37.
    Cugini, J., Scholtz, J.: VISVIP: 3D Visualization of Paths Through Web Sites. In: Proceedings of the International Workshop on Web-Based Information Visualization, pp. 259–263 (1999)Google Scholar
  38. 38.
    Czarkowski, M.: Evaluating Scrutable Adaptive Hypertext. In: Weibelzahl, S., Paramythis, A., Masthoff, J. (eds.) Proceedings of the Fourth Workshop on Empirical Evaluation of Adaptive Systems, held at the 10th International Conference on User Modeling UM2005, Edinburgh, pp. 37–46 (2005)Google Scholar
  39. 39.
    Del Missier, F., Ricci, F.: Understanding Recommender Systems: Experimental Evaluation Challenges. In: Weibelzahl, S., Paramythis, A. (eds.) Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference, pp. 31–40 (2003)Google Scholar
  40. 40.
    Diaper, D. (ed.): Task Analysis for Human-computer Interaction. Ellis Horwood, Chicester (1989)Google Scholar
  41. 41.
    Dimitrova, V.: Using Dialogue Games to Maintain Diagnostic Interactions. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 117–121. Springer, Heidelberg (2003)Google Scholar
  42. 42.
    Dimitrova, V., Self, J., Brna, P.: Applying Interactive Open Learner Models to Learning Technical Terminology. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 148–157. Springer, Heidelberg (2001)Google Scholar
  43. 43.
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction, 2nd edn. Prentice Hall, Englewood Cliffs (1998)Google Scholar
  44. 44.
    Dumas, J.S., Redish, J.C.: A Practical Guide To Usability Testing. Ablex Publishing, Greenwich (1999)Google Scholar
  45. 45.
    Encarnação, L.M., Stoev, S.L.: Application-independent Intelligent User Support System Exploiting Action-sequence Based User Modeling. In: Kay, J. (ed.) User modeling: Proceedings of the Seventh International Conference, UM’99, pp. 245–254. Springer, Heidelberg (1999)Google Scholar
  46. 46.
    Farzan, R., Brusilovsky, P.: Social Navigation Support in a Course Recommendation System. In: Wade, V., Ashman, H., Smyth, B. (eds.) AH 2006. LNCS, vol. 4018, pp. 91–100. Springer, Heidelberg (2006)Google Scholar
  47. 47.
    Fink, J., Kobsa, A., Nill, A.: Information Provision for All Users, Including Disabled and Elderly People. New Review of Hypermedia and Multimedia 4, 163–188 (1998)Google Scholar
  48. 48.
    Fischer, G., Ye, Y.: Personalizing Delivered Information in a Software Reuse Environment. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 178–187. Springer, Heidelberg (2001)Google Scholar
  49. 49.
    Gena, C.: Designing TV Viewer Stereotypes for an Electronic Program Guide. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 247–276. Springer, Heidelberg (2001)Google Scholar
  50. 50.
    Gena, C.: Evaluation Methodologies and User Involvement in User Modeling and Adaptive Systems. Unpub. Phd thesis, Università degli Studi di Torino, Italy (2003)Google Scholar
  51. 51.
    Gena, C.: Methods and Techniques for the Evaluation of User-adaptive Systems. The Knowledge Engineering Review 20(1), 1–37 (2005)Google Scholar
  52. 52.
    Gena, C., Ardissono, L.: Intelligent Support to the Retrieval of Information About Hydric Resources. In: De Bra, P., Nejdl, W. (eds.) AH 2004. LNCS, vol. 3137, pp. 126–135. Springer, Heidelberg (2004)Google Scholar
  53. 53.
    Gena, C., Perna, A., Ravazzi, M.: E-tool: a Personalized Prototype for Web Based Applications. In: de Waard, D., Brookhuis, K., Moraal, J., Toffetti, A. (eds.) Human Factors in Transportation, Communication, Health, and the Workplace, pp. 485–496. Shaker Publishing (2001)Google Scholar
  54. 54.
    Gena, C., Torre, I.: The Importance of Adaptivity to Provide On-Board Services. A Preliminary Evaluation of an Adaptive Tourist Information Service on Board Vehicles. Applied Artificial Intelligence Journal, Special Issue on Mobile A.I.  (2004)Google Scholar
  55. 55.
    Good, N., Schafer, J.B., Konstan, J.A., Botchers, A., Sarwar, B.M., Herlocker, J.L., Ricdl, J.: Combining Collaborative Filtering with Personal Agents for Better Recommendations. In: Proceedings of the Sixteenth National Conference on Artificial Intelligence, pp. 439–446 (1999)Google Scholar
  56. 56.
    Gould, J.D., Lewis, C.: Designing for Usability – Key Principles and What Designers Think. In: Human Factors in Computing Systems, CHI ’83 Proceedings, pp. 50–53. ACM Press, New York (1983)Google Scholar
  57. 57.
    Goy, A., Ardissono, L., Petrone, G.: Personalization in E-Commerce Applications. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization. LNCS, vol. 4321, pp. 485–520. Springer, Heidelberg (2007)Google Scholar
  58. 58.
    Greenbaum, J., Kyng, M. (eds.): Design At Work: Cooperative Design of Computer Systems. Lawrence Erlbaum Associates, Hillsdale (1991)Google Scholar
  59. 59.
    Greenbaum, T.L.: The Handbook of Focus Group Research, 2nd edn. Lexington Books, New York (1998)Google Scholar
  60. 60.
    Habieb-Mammar, H., Tarpin-Bernard, F., Prévôt, P.: Adaptive Presentation of Multimedia Interface Case Study: Brain Story Course. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 15–24. Springer, Heidelberg (2003)Google Scholar
  61. 61.
    Hara, Y., Tomomune, Y., Shigemori, M.: Categorization of Japanese TV Viewers Based on Program Genres They Watch. In: Ardissono, L., Kobsa, A., Maybury, M. (eds.) Personalization and user-adaptive interaction in digital TV, Kluwer Academic Publishers, Dordrecht (2004)Google Scholar
  62. 62.
    Herder, E.: Utility-Based Evaluation of Adaptive Systems. In: Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh, pp. 25–30 (2003)Google Scholar
  63. 63.
    Herder, E., Weinreich, H., Obendorf, H., Mayer, M.: Much to Know About History. In: Wade, V., Ashman, H., Smyth, B. (eds.) AH 2006. LNCS, vol. 4018, pp. 283–287. Springer, Heidelberg (2006)Google Scholar
  64. 64.
    Höök, K.: Evaluating the Utility and Usability of an Adaptive Hypermedia System. In: Proceedings of 1997 International Conference on Intelligent User Interfaces, pp. 179–186. ACM, New York (1997)Google Scholar
  65. 65.
    Höök, K.: Steps to Take Before IUIs Become Real. Journal of Interacting with Computers 12(4), 409–426 (2000)Google Scholar
  66. 66.
    Iglezakis, D.: Is the ACT-value a Valid Estimate for Knowledge? An Empirical Evaluation of the Inference Mechanism of an Adaptive Help System. In: Weibelzahl, S., Paramythis, A., Masthoff, J. (eds.) Proceedings of the Fourth Workshop on Empirical Evaluation of Adaptive Systems, held at the 10th International Conference on User Modeling UM2005, Edinburgh, pp. 19–26 (2005)Google Scholar
  67. 67.
    Ikehara, S., Chin, D.N., Crosby, M.E.: A Model for Integrating an Adaptive Information Filter Utilizing Biosensor Data to Assess Cognitive Load. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 208–212. Springer, Heidelberg (2003)Google Scholar
  68. 68.
    International Organisation for Standardisation ISO 9241-11:1998 Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs), Part 11 Guidance on Usability (1998), http://www.iso.org/
  69. 69.
    International Organisation for Standardisation ISO/TS 16071:2003 Ergonomics of Human-system Interaction - Guidance on Software Accessibility. Technical Specification (2003), http://www.iso.org/
  70. 70.
    Jackson, T., Mathews, E., Lin, D., Olney, A., Graesser, A.: Modeling Student Performance to Enhance the Pedagogy of AutoTutor. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 368–372. Springer, Heidelberg (2003)Google Scholar
  71. 71.
    Jameson, A., Schäfer, R., Weis, T., Berthold, A., Weyrath, T.: Making Systems Sensitive to the User’s Changing Resource Limitations. Knowledge-Based Systems 12(8), 413–425 (1999)Google Scholar
  72. 72.
    Jameson, A., Schwarzkopf, E.: Pros and Cons of Controllability: An Empirical Study. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) AH 2002. LNCS, vol. 2347, pp. 193–202. Springer, Heidelberg (2002)Google Scholar
  73. 73.
    Jameson, A.: Adaptive Interfaces and Agents. In: The Human-Computer Interaction Handbook, pp. 316–318. Lawrence Erlbaum Associates, New Jersey (2003)Google Scholar
  74. 74.
    Jameson, A.: User Modeling Meets Usability Goals. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 1–3. Springer, Heidelberg (2005)Google Scholar
  75. 75.
    Juvina, I., Herder, E.: The Impact of Link Suggestions on User Navigation and User Perception. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 483–492. Springer, Heidelberg (2005)Google Scholar
  76. 76.
    Karagiannidis, C., Sampson, D.: Layered Evaluation of Adaptive Applications and Services. In: Brusilovsky, P., Stock, O., Strapparava, C. (eds.) AH 2000. LNCS, vol. 1892, pp. 343–346. Springer, Heidelberg (2000)Google Scholar
  77. 77.
    Katz-Haas, R.: Ten Guidelines for User-Centered Web Design. Usability Interface 5(1), 12–13 (1998)Google Scholar
  78. 78.
    Kay, J.: Learner Control. User Modeling and User-Adapted Interaction, Tenth. Anniversary Special Issue 11(1-2), 111–127 (2001)MATHGoogle Scholar
  79. 79.
    Keppel, G.: Design and Analysis: A Researcher’s Handbook. Prentice-Hall, Englewood Cliffs (1991)Google Scholar
  80. 80.
    Keppel, G., Saufley, W.H., Tokunaga, H.: Introduction to Design and Analysis, A Student’s Handbook, 2nd edn. Prentice-Hall, Englewood Cliffs (1998)Google Scholar
  81. 81.
    Kirby, M.A.R.: CUSTOM Manual Dpo/std/1.0. Huddersfield: HCI Research Centre, University of Huddersfield (1991)Google Scholar
  82. 82.
    Krogsæter, M., Oppermann, R., Thomas, C.G.: A User Interface Integrating Adaptability and Adaptivity. In: Oppermann, R. (ed.) Adaptive user support, pp. 97–125. Lawrence Erlbaum, Hillsdale (1994)Google Scholar
  83. 83.
    Krug, S.: Don’t Make Me Think! A Common Sense Approach to Web Usability. New Riders Publishing, Indianapolis (2000)Google Scholar
  84. 84.
    Krüger, A., Heckmann, D., Kruppa, M., Wasinger, R.: Adaptive Mobile Guides. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization. LNCS, vol. 4321, pp. 521–549. Springer, Heidelberg (2007)Google Scholar
  85. 85.
    Kuflik, T., Shapira, B., Elovici, Y., Maschiach, A.: Privacy Preservation Improvement By Learning Optimal Profile Generation Rate. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 168–177. Springer, Heidelberg (2003)Google Scholar
  86. 86.
    Kumar, A.: Evaluating Adaptive Generation of Problems in Programming Tutors – Two Studies. In: Proc. of the Fifth Workshop on User-Centred Design and Evaluation of Adaptive Systems, pp. 450–459 (2006)Google Scholar
  87. 87.
    Kurhila, J., Miettinen, M., Nokelainen, P., Tirri, H.: EDUCO - A Collaborative Learning Environment Based on Social Navigation. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) AH 2002. LNCS, vol. 2347, pp. 242–252. Springer, Heidelberg (2002)Google Scholar
  88. 88.
    Lang, T., Secic, M.: How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors and Reviewers. American College of Physicians, Philadelphia (1997)Google Scholar
  89. 89.
    Lavie, T., Meyer, J., Bengler, K., Coughlin, J.F.: The Evaluation of In-Vehicle Adaptive Systems. In: Weibelzahl, S., Paramythis, A., Masthoff, J. (eds.) Proceedings of the Fourth Workshop on Empirical Evaluation of Adaptive Systems, held at the 10th International Conference on User Modeling UM2005, Edinburgh, pp. 9–18 (2005)Google Scholar
  90. 90.
    Lee, J., Lai, K.-Y.: What’s in Design Rationale? Human-Computer Interaction: special issue on design rationale 6(3-4), 251–280 (1991)Google Scholar
  91. 91.
    Litman, D.J., Pan, S.: Predicting and Adapting to Poor Speech Recognition in a Spoken Dialogue System. In: Proceedings of the Seventeenth National Conference on Artificial Intelligence, Austin, TX, pp. 722–728 (2000)Google Scholar
  92. 92.
    Magnini, B., Strapparava, C.: Improving User Modelling with Content-based Techniques. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, Springer, Heidelberg (2001)Google Scholar
  93. 93.
    Magoulas, G.D., Chen, S.Y., Papanikolaou, K.A.: Integrating Layered and Heuristic Evaluation for Adaptive Learning Environments. In: Weibelzahl, S., Paramythis, A. (eds.) Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh, pp. 5–14 (2003)Google Scholar
  94. 94.
    Martin, B., Mitrovic, T.: WETAS: A Web-Based Authoring System for Constraint-Based ITS. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) AH 2002. LNCS, vol. 2347, pp. 396–405. Springer, Heidelberg (2002)Google Scholar
  95. 95.
    Martin, B., Mitrovic, T.: The Effect of Adapting Feedback Generality in ITS E-Learning and Personalization. In: Wade, V., Ashman, H., Smyth, B. (eds.) AH 2006. LNCS, vol. 4018, pp. 192–202. Springer, Heidelberg (2006)Google Scholar
  96. 96.
    Masthoff, J.: The Evaluation of Adaptive Systems. In: Patel, N.V. (ed.) Adaptive evolutionary information systems, pp. 329–347. Idea Group publishing, Hershey (2002)Google Scholar
  97. 97.
    Masthoff, J.: The User As Wizard. In: Proc. of the Fifth Workshop on User-Centred Design and Evaluation of Adaptive Systems, pp. 460–469 (2006)Google Scholar
  98. 98.
    Matsuo, Y.: Word Weighting Based on User’s Browsing History. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 35–44. Springer, Heidelberg (2003)Google Scholar
  99. 99.
    Maulsby, D., Greenberg, S., Mander, R.: Prototyping an Intelligent Agent Through Wizard of Oz. In: ACM SIGCHI Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, May, pp. 277–284. ACM Press, New York (1993)Google Scholar
  100. 100.
    Maybury, M., Brusilovsky, P.: The Adaptive Web. Communications of the ACM 45 (2002)Google Scholar
  101. 101.
    McCreath, E., Kay, J.: IEMS: Helping Users Manage Email. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 263–272. Springer, Heidelberg (2003)Google Scholar
  102. 102.
    McNee, S.M., Lam, S.K, Konstan, J.A., Riedl, J.: Interfaces for Eliciting New User Preferences in Recommender Systems. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 178–187. Springer, Heidelberg (2003)Google Scholar
  103. 103.
    Mitrovic, A.: Investigating Students’ Self-assessment Skills. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 247–250. Springer, Heidelberg (2001)Google Scholar
  104. 104.
    Mitrovic, A., Martin, B.: Evaluating the Effects of Open Student Models on Learning. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) AH 2002. LNCS, vol. 2347, pp. 296–305. Springer, Heidelberg (2002)Google Scholar
  105. 105.
    Mobasher, R.: Data Mining for Web Personalisation. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization. LNCS, vol. 4321, pp. 90–136. Springer, Heidelberg (2007)Google Scholar
  106. 106.
    Mobasher, R., Cooley, R., Srivastava, J.: Automatic Personalization Based on Web Usage Mining. Communications of the ACM 43(8) (2000)Google Scholar
  107. 107.
    Monk, A., Wright, P., Haber, J., Davenport, L.: Improving Your Human Computer Interface: A Practical Approach. BCS Practitioner Series. Prentice-Hall International, Hemel Hempstead (1993)Google Scholar
  108. 108.
    Müller, C., Großmann-Hutter, B., Jameson, A., Rummer, R., Wittig, F.: Recognizing Time Pressure and Cognitive Load on the Basis of Speech: An Experimental Study. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 24–33. Springer, Heidelberg (2001)Google Scholar
  109. 109.
    Nielsen, J.: Usability Engineering. Academic Press, Boston (1993)MATHGoogle Scholar
  110. 110.
    Nielsen, J.: Why You Only Need to Test With 5 Users. In: Alertbox (2000), http://www.useit.com/alertbox/20000319.html
  111. 111.
    Nielsen, J.: Risks of Quantitative Studies. In: Alertbox (2004), http://www.useit.com/alertbox/20040301.html
  112. 112.
    Nielsen, J., Mack, R.L. (eds.): Usability Inspection Methods. John Wiley & Sons, New York (1994)Google Scholar
  113. 113.
    Nielsen, J., Molich, R.: Heuristic Evaluation of User Interfaces. In: Proceedings of CHI ’90, Seattle, Washington, pp. 249–256 (1990)Google Scholar
  114. 114.
    Norman, D.A., Draper, S.W.: User Centered System Design: New Perspective on HCI. Lawrence Erlbaum, Hillsdale (1986)Google Scholar
  115. 115.
    Paramythis, A., Totter, A., Stephanidis, C.: A Modular Approach to the Evaluation of Adaptive User Interfaces. In: Weibelzahl, S., Chin, D., Weber, G. (eds.) Proceedings of the First Workshop on Empirical Evaluation of Adaptive Systems, Sonthofen, Germany, pp. 9–24 (2001)Google Scholar
  116. 116.
    Paramythis, A., Weibelzahl, S.: A Decomposition Model for the Layered Evaluation of Interactive Adaptive Systems. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 438–442. Springer, Heidelberg (2005)Google Scholar
  117. 117.
    Paganelli, L., Paterno’, F.: Intelligent Analysis of User Interactions with Web Applications. In: Proceedings of the 2002 International Conference on Intelligent User Interfaces, ACM Press, New York (2002)Google Scholar
  118. 118.
    Pazzani, M.J., Billsus, D.: Content-based Recommendation Systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization. LNCS, vol. 4321, pp. 324–341. Springer, Heidelberg (2007)Google Scholar
  119. 119.
    Perkowitz, M., Etzioni, O.: Adaptive Web Sites. Communications of the ACM 43(8) (2000)Google Scholar
  120. 120.
    Pierrakos, D., Paliouras, G., Papatheodorou, C., Spyropoulos, C.D.: Web Usage Mining As a Tool for Personalization: a Survey. International Journal of User Modeling and User-Adapted Interaction 13(4), 311–372 (2003)Google Scholar
  121. 121.
    Pirolli, P., Fu, W.T.: SNIF-ACT: A Model of Information Foraging on the World Wide Web. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 45–54. Springer, Heidelberg (2003)Google Scholar
  122. 122.
    Pohl, W.: LaboUr-machine Learning for User Modeling. In: Smith, M.J., Salvendy, G., Koubek, R.J. (eds.) Design of computing systems: Social and ergonomic considerations, vol. B, pp. 27–30. Elsevier, Amsterdam (1997)Google Scholar
  123. 123.
    Pohl, W.: User-adapted Interaction, User Modeling, and Machine Learning. In: Timm, U.J., Rössel, M. (eds.) Proceedings of the sixth german workshop on adaptivity and user modeling in interactive systems, ABIS98, Erlangen (1998)Google Scholar
  124. 124.
    Polson, P.G., Lewis, C., Rieman, J., Wharton, C.: Cognitive Walkthroughs: A Method for Theory- Based Evaluation of User Interfaces. International Journal of Man-Machine Studies 36, 741–773 (1992)Google Scholar
  125. 125.
    Preece, J., Rogers, Y., Sharp, H., Benyon, D.: Human-computer Interaction. Addison-Wesley, Reading (1994)Google Scholar
  126. 126.
    Resnick, P., Varian, H.R.: Special Issue on Recommender Systems. Communications of the ACM 40 (1997)Google Scholar
  127. 127.
    Ritter, F.E., Shadbolt, N., Elliman, D., Young, R., Gobet, F., Baxter, G.: Techniques for Modeling Human and Organizational Behaviour in Synthetic Environments: A Supplementary Review. Wright-Patterson Air Force Base, OH: Human Systems Information Analysis Center (2002)Google Scholar
  128. 128.
    Rizzo, P., Lee, I.H., Shaw, E., Johnson, W.L., Wang, N., Mayer, R.E.: A Semi-Automated Wizard of Oz Interface for Modeling Tutorial Strategies. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 174–178. Springer, Heidelberg (2005)Google Scholar
  129. 129.
    Rosenfeld, L., Morville, P.: Information Archtecture. O’Reilly, Sebastopol (1998)Google Scholar
  130. 130.
    Rubin, J.: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 1st edn. John Wiley & Sons, Chichester (1994)Google Scholar
  131. 131.
    Jr. Santos, E.H., Nguyen, Q., Zhao, E.: Pukinskis: Empirical Evaluation of Adaptive User Modeling in a Medical Information Retrieval Application. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 292–296. Springer, Heidelberg (2003)Google Scholar
  132. 132.
    Sarwar, B.M., Konstan, J.A., Borchers, A., Herlocker, J., Miller, B., Riedl, J.: Using Filtering Agents to Improve Prediction Quality in the GroupLens Research Collaborative Filtering System. In: Proceeding of the ACM Conference on Computer Supported Cooperative Work (CSCW), Seattle, WA, pp. 439–446. ACM Press, New York (1998)Google Scholar
  133. 133.
    Scarano, V., Barra, M., Maglio, P., Negro, A.: GAS: Group Adaptive Systems. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) AH 2002. LNCS, vol. 2347, pp. 47–57. Springer, Heidelberg (2002)Google Scholar
  134. 134.
    Sears, A., Shneiderman, B.: Split Menus: Effectively Using Selection Frequency to Organize Menus. ACM Transactions on Computer-Human Interaction 1(1), 27–51 (1994)Google Scholar
  135. 135.
    Semeraro, G., Ferilli, S., Fanizzi, N., Abbattista, F.: Learning Interaction Models in a Digital Library Service. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 44–53. Springer, Heidelberg (2001)Google Scholar
  136. 136.
    Shardanand, U., Maes, P.: Social Information Filtering for Automating ”Word of Mouth”. In: Proceedings of CHI-95, Denver, CO, pp. 210–217 (1995)Google Scholar
  137. 137.
    Smyth, B., Cotter, P.: Personalized Adaptive Navigation for Mobile Portals. In: Proceedings of the 15th European Conference on Artificial Intelligence - Prestigious Applications of Intelligent Systems, Lyons, France (2002)Google Scholar
  138. 138.
    Specht, M., Kobsa, A.: Interaction of Domain Expertise and Interface Design. In: Adaptive Educational Hypermedia, Workshops on Adaptive Systems and User Modeling on the World Wide Web at WWW-8, Toronto, Canada, and Banff, Canada (1999)Google Scholar
  139. 139.
    Spiliopoulou, M.: Web Usage Mining for Web Site Evaluation. Communications of the ACM 43(8) (2000)Google Scholar
  140. 140.
    Spradley, J.: Participant Observation. Wadsworth Publishing (1980)Google Scholar
  141. 141.
    Stephanidis, C.: Adaptive Techniques for Universal Access. User Modeling and User-Adapted Interaction 11(Issue 1-2), 159–179 (2001)MATHGoogle Scholar
  142. 142.
    Strauss, A.L., Corbin, J.M.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE, Thousand Oaks (1998)Google Scholar
  143. 143.
    Tasso, C., Omero, P.: La Personalizzazione Dei Contenuti Web. Franco Angeli, Milano (2002)Google Scholar
  144. 144.
    Tobar, C.M.: Yet Another Evaluation Framework. In: Weibelzahl, S., Paramythis, A. (eds.) Proceedings of the Second Workshop on Empirical Evaluation of Adaptive Systems, held at the 9th International Conference on User Modeling UM2003, Pittsburgh, pp. 15–24 (2003)Google Scholar
  145. 145.
    Totterdell, P., Boyle, E.: The Evaluation of Adaptive Systems. In: Browne, D., Totterdell, P., Norman, M. (eds.) Adaptive User Interfaces, pp. 161–194. Academic Press, London (1990)Google Scholar
  146. 146.
    Travassos, G.H., Barros, M.O.: Contributions of In Virtuo and In Silico Experiments for the Future of Empirical Studies in Software Engineering. In: Jedlitschka, A., Ciolkowski, M. (eds.) Proceedings of the Second Workshop on the Future of Empirical Studies in Software Engineering, pp. 109–121. Roman Castles, Italy (2003)Google Scholar
  147. 147.
    Uruchurtu, E., MacKinnon, L., Rist, R.: User Cognitive Style and Interface Design for Personal, Adaptive Learning. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 154–163. Springer, Heidelberg (2005)Google Scholar
  148. 148.
    van Solingen, R., Berghout, E.: The Goal/question/metric Method: A Practical Guide for Quality Improvement of Software Development. McGraw-Hill, London (1999)Google Scholar
  149. 149.
    van Barneveld, J., van Setten, M.: Involving users in the design of user interfaces for TV recommender systems. 3rd Workshop on Personalization in Future TV, Associated with UM03, Johnstown, PA (2003)Google Scholar
  150. 150.
    Web Accessibility Initiative W3C - Evaluating Web Sites for Accessibility, http://www.w3.org/WAI/eval/
  151. 151.
    Web Accessibility Initiative W3C - Web Content Accessibility Guidelines 1.0, http://www.w3c.org/TR/WAI-WEBCONTENT
  152. 152.
    Weibelzahl, S.: Evaluation of Adaptive Systems. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 292–294. Springer, Heidelberg (2001)Google Scholar
  153. 153.
    Weibelzahl, S.: Evaluation of Adaptive Systems. Dissertation. University of Trier, Germany (2003)Google Scholar
  154. 154.
    Weibelzahl, S.: Problems and pitfalls in the evaluation of adaptive systems. In: Chen, S., Magoulas, G. (eds.) Adaptable and Adaptive Hypermedia Systems, pp. 285–299. IRM Press, Hershey (2005)Google Scholar
  155. 155.
    Weibelzahl, S., Lauer, C.U.: Framework for the Evaluation of Adaptive CBRSystems. In: Reimer, U., Schmitt, S., Vollrath, I. (eds.) Proceedings of the 9th German Workshop on Case-Based Reasoning (GWCBR01), Aachen, pp. 254–263. Shaker (2001)Google Scholar
  156. 156.
    Weibelzahl, S., Weber, G.: A database of empirical evaluations of adaptive systems. In: Klinkenberg, R., Rüping, S., Fick, A., Henze, N., Herzog, C., Molitor, R., Schröder, O. (eds.) Proceedings of Workshop Lernen - Lehren - Wissen - Adaptivität (LLWA 01), Research Report in Computer Science Nr. 763, University of Dortmund, pp. 302–306 (2001)Google Scholar
  157. 157.
    Weibelzahl, S., Jedlitschka, A., Ayari, B.: Eliciting Requirements for a Adaptive Decision Support System through Structured User Interviews. In: Proc. of the Fifth Workshop on User-Centred Design and Evaluation of Adaptive Systems, pp. 470–478 (2006)Google Scholar
  158. 158.
    Weibelzahl, S., Weber, G.: Evaluating the inference mechanism of adaptive learning systems. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 154–162. Springer, Heidelberg (2003)Google Scholar
  159. 159.
    Whiteside, J., Bennett, J., Holtzblatt, K.: Usability Engineering: Our Experience and Evolution. In: Helander, M. (ed.) Handbook of Human-Computer Interaction, pp. 791–817. North-Holland, New York (1988)Google Scholar
  160. 160.
    Wilkinson, L. and Task Force on Statistical Inference: Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 54(8), 594–604 (1999)Google Scholar
  161. 161.
    Wingender, E. (ed.): In Silico Biology. An international journal on computational molecular biology. IOS Press, Amsterdam (1998)Google Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Cristina Gena
    • 1
  • Stephan Weibelzahl
    • 2
  1. 1.Dipartimento di Informatica, Università di Torino, Corso Svizzera 185, TorinoItaly
  2. 2.School of Informatics, National College of Ireland, Mayor Street, DublinIreland

Personalised recommendations