User Modeling and User-Adapted Interaction

, Volume 20, Issue 5, pp 383–453 | Cite as

Layered evaluation of interactive adaptive systems: framework and formative methods

  • Alexandros Paramythis
  • Stephan Weibelzahl
  • Judith Masthoff
Original Paper

Abstract

The evaluation of interactive adaptive systems has long been acknowledged to be a complicated and demanding endeavour. Some promising approaches in the recent past have attempted tackling the problem of evaluating adaptivity by “decomposing” and evaluating it in a “piece-wise” manner. Separating the evaluation of different aspects can help to identify problems in the adaptation process. This paper presents a framework that can be used to guide the “layered” evaluation of adaptive systems, and a set of formative methods that have been tailored or specially developed for the evaluation of adaptivity. The proposed framework unifies previous approaches in the literature and has already been used, in various guises, in recent research work. The presented methods are related to the layers in the framework and the stages in the development lifecycle of interactive systems. The paper also discusses practical issues surrounding the employment of the above, and provides a brief overview of complementary and alternative approaches in the literature.

Keywords

Layered evaluation Evaluation framework Formative evaluation methods Design 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ackerman, M.S., Cranor, L.F., Reagle, J.: Privacy in e-commerce: examining user scenarios and privacy preferences. In: 1st ACM Conference on Electronic Commerce, pp. 1–8. ACM, Denver, CO (1999)Google Scholar
  2. Albrecht D., Zuckerman I.: Statistical and probabilistic methods for user modeling. Special issue User Model. User-Adap. Inter. 17(1–2), 1–215 (2007)CrossRefGoogle Scholar
  3. Anderson, J.R., Boyle, C.F., Yost, G.: The geometry tutor. In: 9th International Joint Conference on Artificial Intelligence, Los Angeles, CA, pp. 1–7. Morgan Kaufmann, San Francisco (1985)Google Scholar
  4. Arruabarrena, R., Pérez, T., López-Cu adrado, J., Gutiérrez, J., Vadillo, J.: On evaluating adaptive systems for education. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Malaga, Spain. LNCS, vol. 2347, pp. 363–367. Springer, Berlin (2002)Google Scholar
  5. Batliner A., Steidl S., Hacker C., Nöth E.: Private emotions versus social interaction: a data-driven approach towards analysing emotion in speech. User Model. User-Adap. Inter. 18(1-2), 175–206 (2008)CrossRefGoogle Scholar
  6. Berkovsky S., Kuflik T., Ricci F.: Mediation of user models for enhanced personalization in recommender systems. User Model. User-Adap. Inter. 18(3), 245–286 (2008)CrossRefGoogle Scholar
  7. Beyer H., Holtzblatt K.: Contextual Design: Defining Customer-Centred Systems. Morgan Kaufmann, San Francisco (1997)Google Scholar
  8. Billsus, D., Pazzani, M.: A hybrid user model for news story classification. In: 7th International Conference on User Modeling, Banff, Canada, pp. 98–108. Springer, Vienna (1999)Google Scholar
  9. Bontcheva K., Wilks Y.: Tailoring automatically generated hypertext. User Model. User-Adap. Inter. 15(1-2), 135–168 (2005)CrossRefGoogle Scholar
  10. Boyle C., Encarnacion A.O.: MetaDoc: an adaptive hypertext reading system. User Model. User-Adap. Inter. 4(1), 1–19 (1994)CrossRefGoogle Scholar
  11. Brown E.J., Brailsford T.J., Fisher T., Moore A.: Evaluating learning style personalization in adaptive systems: quantitative methods and approaches. IEEE Trans. Learn. Technol. 2(1), 10–22 (2009)CrossRefGoogle Scholar
  12. Browne D., Norman M., Riches D.: Why build adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 15–57. Academic Press, London (1990)Google Scholar
  13. Brusilovsky P., Eklund J.: A study of user model based link annotation in educational hypermedia. J. Univ. Comput. Sci. 4(4), 429–448 (1998)Google Scholar
  14. Brusilovsky, P., Karagiannidis, C., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 1–8. Sonthofen, Germany (2001)Google Scholar
  15. Brusilovsky, P., Farzan, R., Ahn, J.: Layered evaluation of adaptive search. In: Workshop on Evaluating Exploratory Search Systems at SIGIR06, pp. 11–13. Seattle, WA (2006)Google Scholar
  16. Bull S., Kay J.: Student models that invite the learner in: the SMILI Open learner modelling framework. Int. J. Artif. Intell. Edu. 17(2), 89–120 (2007)Google Scholar
  17. Card S.K., Thomas T.P., Newell A.: The Psychology of Human–Computer Interaction. Lawrence Erbaum Associates, London (1983)Google Scholar
  18. Carmagnola F., Cena F., Console L., Cortassa O., Gena C., Goy A., Torre I., Toso A., Vernero F.: Tag-based user modeling for social multi-device adaptive guides. User Model. User-Adap. Inter. 18(5), 497–538 (2008)CrossRefGoogle Scholar
  19. Carmichael D.J., Kay J., Kummerfeld B.: Consistent modelling of users, devices and sensors in a ubiquitous computing environment. User Model. User-Adap. Inter. 15(3-4), 197–234 (2005)CrossRefGoogle Scholar
  20. Carroll M.J.: Five reasons for scenario-based design. Interact. Comput. 13(1), 43–60 (2000)CrossRefGoogle Scholar
  21. Cena F., Console L., Gena C., Goy A., Levi G., Modeo S., Torre I.: Integrating heterogeneous adaptation techniques to build a flexible and usable mobile tourist guide. AI Commun. 19(4), 369–384 (2006)MATHMathSciNetGoogle Scholar
  22. Chen H.T.: A comprehensive typology for program evaluation. Am. J. Eval. 17(2), 121–130 (1996)CrossRefGoogle Scholar
  23. Cheverst K., Byun H. E., Fitton D., Sas C., Kray C., Villar N.: Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Model. User-Adap. Inter. 15(3-4), 235–273 (2005)CrossRefGoogle Scholar
  24. Chickering D.M., Paek T.: Personalizing influence diagrams: applying online learning strategies to dialogue management. User Model. User-Adap. Inter. 17(1-2), 71–91 (2007)CrossRefGoogle Scholar
  25. Chin D.: Empirical evaluation of user models and user-adapted systems. User Model. User-Adap. Inter. 11(1-2), 181–194 (2001)MATHCrossRefGoogle Scholar
  26. Claypool, M., Le, P., Wased, M., Brown, D.: Implicit interest indicators. In: International Conference on Intelligent User Interfaces, pp. 33–40. ACM, Santa Fe, NM (2001)Google Scholar
  27. Conati C., MacLaren H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adap. Inter. 19(3), 267–303 (2009)CrossRefGoogle Scholar
  28. Cooper A.: The Inmates are Running the Asylum. Macmillan, Indianapolis (1999)Google Scholar
  29. Cooper, D., Arroyo, I., Woolf, B., Muldner, K., Burleson, W., Christopherson, R.: Sensors model student self concept in the classroom. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 30–41. Springer, Berlin (2009)Google Scholar
  30. Cramer H., Evers V., Ramlal S., van Someren M., Rutledge L., Stash N., Aroyo L., Wielinga B.: The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User-Adap. Inter. 18(5), 455–496 (2008)CrossRefGoogle Scholar
  31. Cronbach L.J., Ambron S.R., Dornbusch S.M., Hess R.D., Hornik R.C., Philips D.C., Walker D.F., Weiner S.S.: Toward Reform of Program Evaluation. Jossey-Bass, San Francisco (1980)Google Scholar
  32. Czarkowski, M.: A scrutable adaptive hypertext. PhD Thesis, University of Sydney (2006)Google Scholar
  33. Damiano R., Gena C., Lombardo V., Nunnari F., Pizzo A.: A stroll with Carletto: adaptation in drama-based tours with virtual characters. User Model. User-Adap. Inter. 18(5), 417–453 (2008)CrossRefGoogle Scholar
  34. De Bra, P., Houben, G., Wu, H.: AHAM: a Dexter-based reference model for adaptive hypermedia. In: 10th ACM Conference on Hypertext and Hypermedia, pp. 147–156. ACM, Darmstadt, Germany (1999)Google Scholar
  35. de Campos L., Fernández-Luna J., Huete J., Rueda-Morales M.: Managing uncertainty in group recommending processes. User Model. User-Adap. Inter. 19(3), 207–242 (2009)CrossRefGoogle Scholar
  36. de Rosis, F., Mazzotta, I., Miceli, M., Poggi, I.: Persuasion artifices to promote wellbeing. In: 1st International Conference on Persuasive Technology, Eindhoven, The Netherlands. LNCS, vol. 3962, pp. 84–95. Springer, Berlin (2006)Google Scholar
  37. Degemmis M., Lops P., Semeraro G.: A content-collaborative recommender that exploits WordNet-based user profiles for neighborhood formation. User Model. User-Adap. Inter. 17(3), 217–255 (2007)CrossRefGoogle Scholar
  38. Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: Workshop on the What, Who, Where, When, and How of Context-Awareness at CHI 2000, pp. 304–307. The Hague, The Netherlands (2000)Google Scholar
  39. Dix A., Finlay J., Abowd G., Beale R.: Human Computer Interaction. 2nd edn. Prentice-Hall, Englewood Cliffs (1998)Google Scholar
  40. D’Mello S.K., Craig S.D., Sullins J., Graesser A.C.: Predicting affective states through an emote-aloud procedure from AutoTutor’s mixed-initiative dialogue. Int. J. Artif. Intell. Edu. 16(1), 3–28 (2006)Google Scholar
  41. D’Mello S.K., Craig S.D., Witherspoon A., McDaniel B., Graesser A.C.: Automatic detection of learner’s affect from conversational cues. User Model. User-Adap. Inter. 18(1-2), 45–80 (2008)CrossRefGoogle Scholar
  42. Domshlak C., Joachims T.: Efficient and non-parametric reasoning over user preferences. User Model. User-Adap. Inter. 17(1-2), 41–69 (2007)CrossRefGoogle Scholar
  43. Dumas J.S., Loring B.A.: Moderating Usability Tests: Principles and Practices for Interacting. Morgan Kaufmann, San Francisco (2008)Google Scholar
  44. Encarnação, L., Stoev, S.: Application-independent intelligent user support system exploiting action-sequence based user modeling. In: 7th International Conference on User Modeling, pp. 245–254. Banff, Canada (1999)Google Scholar
  45. Ericsson, K.A., Simon, H.A.: Protocol analysis: verbal reports as data, revised edition. MIT Press, Cambridge, MA (1993)Google Scholar
  46. Forbes-Riley K., Rotaru M., Litman D.J.: The relative impact of student affect on performance models in a spoken dialogue tutoring system. User Model. User-Adap. Inter. 18(1-2), 11–43 (2008)CrossRefGoogle Scholar
  47. Gena C.: Methods and techniques for the evaluation of user-adaptive systems. Knowl. Eng. Rev. 20(1), 1–37 (2005)CrossRefGoogle Scholar
  48. Gena, C., Ardissono, L.: Intelligent support to the retrieval of information about hydric resources. In: 3rd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems. LNCS, vol. 3137, pp. 126–135. Springer, Berlin (2004)Google Scholar
  49. Gena C., Weibelzahl S.: Usability engineering for the adaptive web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 720–762. Springer, Berlin (2007)Google Scholar
  50. George S., Zukerman I., Niemann M.: Inferences, suppositions and explanatory extensions in argument interpretation. User Model. User-Adap. Inter. 17(5), 439–474 (2007)CrossRefGoogle Scholar
  51. Glahn, C., Specht, M., Koper, R.: Smart indicators on learning interactions. In: 2nd European Conference on Technology Enhanced Learning, Crete, Greece. LNCS, vol. 4753, pp. 56–70. Springer, Berlin (2007)Google Scholar
  52. Goecks, J., Shavlik, J.: Learning users’ interests by unobtrusively observing their normal behavior. In: 5th International Conference on Intelligent User Interfaces, pp. 129–132. ACM, New Orleans, LA (2000)Google Scholar
  53. Goren-Bar D., Graziola I., Pianesi F., Zancanaro M.: The influence of personality factors on visitor attitudes towards adaptivity dimensions for mobile museum guides. User Model. User-Adap. Inter. 16(1), 31–62 (2006)CrossRefGoogle Scholar
  54. Goren-Bar, D., Graziola, I., Rocchi, C., Pianesi, F., Stock, O., Zancanaro, M.: Designing and redesigning an affective interface for an adaptive museum guide. In: 1st International Conference on Affective Computing and Intelligent Interaction, Beijing, China. LNCS, vol. 3784, pp. 939–946. Springer, Berlin (2005)Google Scholar
  55. Gould J.D., Lewis C.: Designing for usability: key principles and what designers think. Commun. ACM 28(3), 300–311 (1985)CrossRefGoogle Scholar
  56. Gould, J., Conti, J., Hovanyecz, T.: Composing letters with a simulated listening typewriter. In: 1st ACM Conference on Human Factors in Computer Systems (CHI), pp. 367–370. ACM, Gaithersburg, MD (1982)Google Scholar
  57. Green P.D., Ha E.S., Bullock G.J.: Enough already about “black box” experiments: Studying mediation is more difficult than most scholars suppose. Ann. Am. Acad. Political Soc. Sci. 628(1), 200–208 (2010)Google Scholar
  58. Grudin, J., Pruitt, J.: Personas, participatory design, and product development: An infrastructure for engagement. In: Participatory Design Conference, pp. 144–161, ACM, Malmö, Sweden (2002)Google Scholar
  59. Guzmán E., Conejo R., Pérez de la Cruz J.-L.: Adaptive testing for hierarchical student models. User Model. User-Adap. Inter. 17(1-2), 119–157 (2007)CrossRefGoogle Scholar
  60. Herder, E.: Utility-based evaluation of adaptive systems. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 25–30. Johnstown, PA, USA (2003)Google Scholar
  61. Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)CrossRefGoogle Scholar
  62. Hertzum M., Hansen K., Andersen H.H.K.: Scrutinizing usability evaluation: does thinking aloud affect behaviour and mental workload?. Behav. Inf. Technol. 28(2), 165–181 (2009)CrossRefGoogle Scholar
  63. Hollink V., van Someren M., Wielinga B.: Discovering stages in web navigation for problem-oriented navigation support. User Model. User-Adap. Inter. 17(1-2), 183–214 (2007)CrossRefGoogle Scholar
  64. Höök K.: Steps to take before intelligent user interfaces become real. Interact. Comput. 12(4), 409–426 (2000)CrossRefGoogle Scholar
  65. Hoppe, H., Tauber, M., Ziegler, J.: A survey of models and formal description methods in HCI with example applications. In: ESPRIT Project, vol. 385 (1986)Google Scholar
  66. Horvitz E., Paek T.: Complementary computing: Policies for transferring callers from dialog systems to human receptionists. User Model. User-Adap. Inter. 17(1-2), 159–182 (2007)CrossRefGoogle Scholar
  67. Jameson A.: Systems that Adapt to their Users: An Integrative Perspective. Saarland University, Saarbrücken (2001)Google Scholar
  68. Jameson A.: Adaptive interfaces and agents. In: Jacko, J.A., Sears, A. (eds) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 305–330. L. Erlbaum Associates, Hillsdale (2003)Google Scholar
  69. Jameson, A.: User modeling meets usability goals. In: 10th International Conference on User Modeling, Edinburgh, UK. LNAI, vol. 3538, pp. 1–3. Springer, Berlin (2005)Google Scholar
  70. Jameson A.: Adaptive user interfaces and agents. 2nd edn. In: Sears, A., Jacko, J. (eds) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 433–458. CRC Press, Boca Raton (2008)Google Scholar
  71. Jameson A.: Understanding and dealing with usability side effects of intelligent processing. AI Mag. 30(4), 23–40 (2009)Google Scholar
  72. Jameson A., Schwarzkopf, E.: Pros and cons of controllability: An empirical study. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 193-202. Springer, Berlin (2002)Google Scholar
  73. Kaplan C., Fenwick J., Chen J.: Adaptive hypertext navigation based on user goals and context. User Model. User-Adap. Inter. 3(3), 193–220 (1993)CrossRefGoogle Scholar
  74. Karagiannidis, C., Sampson, D.: Layered evaluation of adaptive applications and services. In: 1st International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Trento, Italy. LNCS, vol. 1892, pp. 343–346. Springer, Berlin, (2000)Google Scholar
  75. Kay, J.: Stereotypes, student models and scrutability. 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1839, pp. 19–30. Springer, Berlin, (2000)Google Scholar
  76. Kay J.: Learner control. User Model. User-Adap. Inter. 11(1-2), 111–127 (2001)MATHCrossRefGoogle Scholar
  77. Knutov E., De Bra P., Pechenizkiy M.: AH—12 years later: a comprehensive survey of adaptive hypermedia methods and techniques. New Rev. Hyperme’d. Multime’d. 15(1), 5–38 (2009)CrossRefGoogle Scholar
  78. Kobsa A.: Privacy-enhanced web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 628–670. Springer, Berlin (2007)Google Scholar
  79. Koch, N., Wirsing, M.: The Munich reference model for adaptive hypermedia applications. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 213–222. Springer, Berlin (2002)Google Scholar
  80. Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: 14th International Joint Conference on Artificial Intelligence, Montréal, Canada, pp. 1137–1145. Morgan Kaufmann, San Francisco (1995)Google Scholar
  81. Kosba E., Dimitrova V., Boyle R.: Adaptive feedback generation to support teachers in web-based distance education. User Model. User-Adap. Inter. 17(4), 379–413 (2007)CrossRefGoogle Scholar
  82. Krogsæter M., Oppermann R., Thomas G.C.: A user interface integrating adaptability and adaptivity. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 97–125. Lawrence Erlbaum, Hillsdale, NJ (1994)Google Scholar
  83. Krueger R., Casey M.: Focus Groups: A Practical Guide for Applied Research. 4th edn. Sage Publications, Los Angeles (2009)Google Scholar
  84. Kruppa, M., Aslan, I.: Parallel presentations for heterogeneous user groups—an initial user study. In: 4th International Conference on Intelligent Technologies for Interactive Entertainment, Madonna di Campiglio, Italy. LNAI. vol. 3814, pp. 54–63, Springer, Berlin (2005)Google Scholar
  85. Law A., Freer Y., Hunter J., Logie R., McIntosh N., Quinn J.: A comparison of graphical and textual presentations of time series data to support medical decision making in the neonatal intensive care unit. J. Clin. Monitor. Comput. 19(3), 183–194 (2005)CrossRefGoogle Scholar
  86. Lekakos G., Giaglis G.: A hybrid approach for improving predictive accuracy of collaborative filtering algorithms. User Model. User-Adap. Inter. 17(1-2), 5–40 (2007)CrossRefGoogle Scholar
  87. Lewis C.: Using the ‘thinking-aloud’ method in cognitive interface design. In: Research Report RC9265. IBM T.J. Watson Research Center, Yorktown Heights, NY (1982)Google Scholar
  88. Ley, T., Kump, B., Maas, A., Maiden, N., Albert, D.: Evaluating the adaptation of a learning system before the prototype is ready: A paper-based lab study. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 331–336, Springer, Berlin (2009)Google Scholar
  89. Limongelli, C., Sciarrone, F., Vaste, G.: LS-Plan: An effective combination of dynamic courseware generation and learning styles in web-based education. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 133–142, Springer, Berlin (2008)Google Scholar
  90. MacLaren, B., Koedinger, K.: When and why does mastery learning work: instructional experiments with ACT-R “SimStudents”. In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France, pp. 355–366. Springer, Berlin (2002)Google Scholar
  91. Magoulas, G.D., Chen, S.Y., Papanikolaou, K.A.: Integrating layered and heuristic evaluation for adaptive learning environments. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 5–14, Johnstown, PA (2003)Google Scholar
  92. Maguire M.: Methods to support human-centred design. Int. J. Hum. Comput. Stud. 55(4), 587–634 (2001)MATHCrossRefGoogle Scholar
  93. Masthoff J.: The evaluation of adaptive systems. In: Patel, N. (ed.) Adaptive Evolutionary Information Systems, pp. 329–347. Idea Group Publishing, London (2002)Google Scholar
  94. Masthoff J.: Group modeling: slecting a sequence of television items to suit a group of viewers. User Model. User-Adap. Inter. 14(1), 37–85 (2004)CrossRefGoogle Scholar
  95. Masthoff, J.: The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. In: 5th Workshop on User-Centred Design and Evaluation of Adaptive Systems at AH06, pp. 460–469. Dublin, Ireland (2006)Google Scholar
  96. Masthoff, J.: Automatically constructing good hierarchies: HCI meets AI, unpublishedGoogle Scholar
  97. Masthoff J., Gatt A.: In pursuit of satisfaction and the prevention of embarrassment: affective state in group recommender systems. User Model. User-Adap. Inter. 16(3-4), 281–319 (2006)CrossRefGoogle Scholar
  98. Masthoff, J., Vasconcelos, W.W., Aitken, C., Correa da Silva, F.S.: Agent-based group modelling for ambient intelligence. In: AISB Symposium on Affective Smart Environments, Newcastle, UK (2007)Google Scholar
  99. Maulsby, D., Greenberg, S., Mander, R.: Prototyping an intelligent agent through wizard of Oz. In: 10th ACM Conference on Human Factors in Computing Systems, pp. 277–284. ACM, Amsterdam, The Netherlands (1993)Google Scholar
  100. McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: How accuracy metrics have hurt recommender systems. In: CHI Work in Progress, Montréal, Canada (2006)Google Scholar
  101. Miettinen M., Oulasvirta A.: Predicting time-sharing in mobile interaction. User Model. User-Adap. Inter. 17(5), 475–510 (2007)CrossRefGoogle Scholar
  102. Millán E., Pérez de la Cruz J.L.: Diagnosis algorithm for student modeling diagnosis and its evaluation. User Model. User-Adap. Inter. 12(2–3), 281–330 (2002)MATHCrossRefGoogle Scholar
  103. Mobasher B.: Data mining for web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 90–135. Springer, Berlin (2007)Google Scholar
  104. : Special issue on data mining for personalization. User Model. User-Adap. Inter. 19(1-2), 1–166 (2009)CrossRefGoogle Scholar
  105. Moncur, W., Masthoff, J., Reiter, E.: What do you want to know? Investigating the information requirements of patient supporters. In: 21th IEEE International Symposium on Computer-Based Medical Systems, pp. 443–448. IEEE, Jyväskylä, Finland (2008)Google Scholar
  106. Murray T.: Formative qualitative evaluation for ‘exploratory’ ITS research. Int. J. Artif. Intell. Educ. 4(2-3), 179–207 (1993)Google Scholar
  107. Nguyen, H., Santos, E. Jr.: An evaluation of the accuracy of capturing user intent for information retrieval. In: International Conference on Artificial Intelligence, pp. 341–350. CSREA Press, Las Vegas, NV (2007)Google Scholar
  108. Nguyen, H., Masthoff, J., Edwards, P.: Modelling a receiver’s position to persuasive arguments. In: 2nd International Conference on Persuasive Technology, Palo Alto, CA. LNCS, vol. 4744, pp. 271–282. Springer, Berlin (2007)Google Scholar
  109. Nielsen J.: Evaluating the thinking-aloud technique for use by computer scientists. In: Hartson H.R., Hix D. (eds.) Advances in Human–Computer interaction, vol.3, pp. 69–82. Ablex, Norwood, NJ (1993)Google Scholar
  110. Nielsen J.: Heuristic evaluation. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 25–64. John Wiley & Sons, New York (1994a)Google Scholar
  111. Nielsen J.: Usability Engineering. 2nd edn. Morgan Kaufmann, San Francisco (1994b)Google Scholar
  112. Norman A.D.: How might people interact with agents. Commun. ACM 37(7), 68–71 (1994)CrossRefGoogle Scholar
  113. Nückles M., Winter A., Wittwer J., Herbert M., Hübner S.: How do experts adapt their explanations to a layperson’s knowledge in asynchronous communication? An experimental study. User Model. User-Adap. Inter. 16(2), 87–127 (2006)CrossRefGoogle Scholar
  114. Ogata K.: Modern Control Engineering. 5th edn. Prentice Hall, Upper Saddle River, NJ (2009)Google Scholar
  115. Ohene-Djan, J.: Ownership transfer via personalisation as a value-adding strategy for web-based education. In: Workshop on Adaptive Systems for Web-Based Education at AH2002, pp. 27–41. Málaga, Spain (2002)Google Scholar
  116. Oliver, N., Horvitz, E.: A comparison of HMMs and dynamic bayesian networks for recognizing office activities. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 199–209. Springer, Berlin (2005)Google Scholar
  117. O’Malley, C.E., Draper, S.W., Riley, M.S.: Constructive interaction: A method for studying Human–Computer-human interaction. In: 1st International Conference on Human–Computer Interaction, pp. 269–274. Honolulu, HI (1984)Google Scholar
  118. Oppermann R.: Adaptively supported adaptability. Int. J. Hum. Comput. Stud. 40(3), 455–472 (1994)CrossRefGoogle Scholar
  119. Oppermann R.: Introduction. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 1–13. Lawrence Erlbaum Associates, Hillsdale (1995)Google Scholar
  120. Ortigosa, A., Carro, R. M.: The continuous empirical evaluation approach: Evaluating adaptive web-based courses. In: 9th International Conference on User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 163–167. Springer, Berlin (2003)Google Scholar
  121. Paek T., Chickering D.M.: Improving command and control speech recognition on mobile devices: using predictive user models for language modelling. User Model. User-Adap. Inter. 17(1-2), 93–117 (2007)CrossRefGoogle Scholar
  122. Paramythis, A., Totter, A., Stephanidis, C.: A modular approach to the evaluation of adaptive user interfaces. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 9–24. Sonthofen, Germany (2001)Google Scholar
  123. Paramythis, A., Weibelzahl, S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 438–442 (2005)Google Scholar
  124. Person, N.K., Graesser, A.C.: Tutoring research group human or computer? AutoTutor in a bystander Turing test. In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France. LNCS, vol. 2363, pp. 821–830. Springer, Berlin (2002)Google Scholar
  125. Petrelli D., Not E.: User-centred design of flexible hypermedia for a mobile guide: reflections on the HyperAudio experience. User Model. User-Adap. Inter. 15(3-4), 303–338 (2005)CrossRefGoogle Scholar
  126. Pohl, W.: LaboUr—Machine learning for user modeling. In: 7th International Conference on Human–Computer Interaction, pp. 27–30. Elsevier, Amsterdam (1997)Google Scholar
  127. Pohl W.: Logic-based representation and reasoning for user modeling shell systems. User Model. User-Adap. Inter. 9(3), 217–282 (1999)MathSciNetCrossRefGoogle Scholar
  128. Popescu, E.: Evaluating the impact of adaptation to learning styles in a web-based educational system. In: 8th International Conference on Web-Based Learning, Aachen, Germany. LNCS, vol. 5686, pp. 343–352. Springer, Berlin (2009)Google Scholar
  129. Porayska-Pomsta K., Mavrikis M., Pain H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User-Adap. Inter. 18(1-2), 125–173 (2008)CrossRefGoogle Scholar
  130. Preece J., Rogers Y., Sharp H., Benyon D.: Human–Computer Interaction. Addison-Wesley, Reading (1994)Google Scholar
  131. Robson C.: Experiment, Design, and Statistics in Psychology. 3rd edn. Penguin, London (1994)Google Scholar
  132. Santos, O., Boticario, J.: Guiding learners in learning management systems through recommendations. In: 4th European Conference on Technology Enhanced Learning, Nice, France. LNCS, vol. 5794, pp. 596–601. Springer, Berlin (2009)Google Scholar
  133. Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: Methods and metrics for cold-start collaborative filtering. In: 25th Annual international ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, pp. 253–260. ACM, New York (2002)Google Scholar
  134. Schmidt, D., Zukerman, I., Albrecht, D.: Assessing the impact of measurement uncertainty on user models in spatial domains. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 210–222. Springer, Berlin (2009)Google Scholar
  135. Scriven M.: Produce evaluation. In: Smith, N.L. (ed.) New Techniques for Evaluation, pp. 121–126. Sage, Beverly Hills (1981)Google Scholar
  136. Scriven M.: Beyond formative and summative evaluation. In: McLaughlin, G.W., Phillips, D.C. (eds) Evaluation and Education: At Quarter Century, pp. 19–64. University of Chicago Press, Chicago (1991)Google Scholar
  137. Scriven M.: Types of evaluation and types of evaluator. Eval. Pract. 17(2), 151–162 (1996)CrossRefGoogle Scholar
  138. Serna A., Pigot H., Rialle V.: Modeling the progression of Alzheimer’s disease for cognitive assistance in smart homes. User Model. User-Adap. Inter. 17(4), 415–438 (2007)CrossRefGoogle Scholar
  139. Shneiderman B.: Designing the User Interface: Strategies for Effective Human–Computer Interaction. Addison Wesley, Reading (1998)Google Scholar
  140. Sixsmith A.J.: An evaluation of an intelligent home monitoring system. J. Telemed. Telecare 6(2), 63–72 (2000)CrossRefGoogle Scholar
  141. Spada, D., Sánchez-Montañés, M., Paredes, P., Carro, R.: Towards inferring sequential-global dimension of learning styles from mouse movement patterns. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 337–340. Springer, Berlin (2008)Google Scholar
  142. Stary, C., Totter, A.: How to integrate concepts of the design and the evaluation of adaptable and adaptive user interfaces. In: 3rd ERCIM Workshop on User Interfaces for All, pp. 68–75. Obernai, France (1997)Google Scholar
  143. Stamou S., Ntoulas A.: Search personalization through query and page topical analysis. User Model. User-Adap. Inter. 19(1), 5–33 (2009)CrossRefGoogle Scholar
  144. Stock O., Zancanaro M.: PEACH—Intelligent Interfaces for Museum Visits. Springer, Berlin (2007)CrossRefGoogle Scholar
  145. Stock O., Zancanaro M., Busetta P., Callaway C., Krüger A., Kruppa M., Kuflik T. et al.: Adaptive, intelligent presentation of information for the museum visitor in PEACH. User Model. User-Adap. Inter. 17(3), 257–304 (2007)CrossRefGoogle Scholar
  146. Suebnukarn S., Haddawy P.: Modeling individual and collaborative problem-solving in medical problem-based learning. User Model. User-Adap. Inter. 16(3-4), 211–248 (2006)CrossRefGoogle Scholar
  147. Tarpin-Bernard, F., Marfisi-Schottman, I., Habieb-Mammar, H.: AnAmeter: The first steps to evaluating adaptation. In: 6th Workshop on User-Centred Design and Evaluation of Adaptive Systems at UMAP2009, pp. 11–20. CEUR, Trento, Italy (2009)Google Scholar
  148. Tintarev, N., Masthoff, J.: Effective explanations of recommendations: User-centered design. In: ACM conference on Recommender systems, pp. 153–156. ACM, Minneapolis, MN (2007)Google Scholar
  149. Tintarev, N., Masthoff, J.: The effectiveness of personalized movie explanations: An experiment using commercial meta-data. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 204–213. Springer, Berlin (2008)Google Scholar
  150. Tintarev, N., Masthoff, J.: Evaluating recommender explanations: problems experienced and lessons learned for the evaluation of adaptive systems. In: 6th Workshop on User Centered Design and Evaluation at UMAP09, pp. 54–63. CEUR, Trento, Italy (2009)Google Scholar
  151. Tobar, C.M.: Yet another evaluation framework. In: Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 15–24. Johnstown, PA (2003)Google Scholar
  152. Totterdell P., Boyle E.: The evaluation of adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 161–194. Academic Press, London (1990)Google Scholar
  153. Totterdell P., Rautenbach P.: Adaptation as a problem of design. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 61–84. Academic Press, London (1990)Google Scholar
  154. Totterdell P., Rautenbach P., Wilkinson A., Anderson S.: Adaptive interface techniques. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 131–160. Academic Press, London (1990)Google Scholar
  155. Trewin, S.: Configuration agents, control and privacy. In: ACM Conference on Universal Usability, pp. 9–16. ACM, Arlington, VA (2000)Google Scholar
  156. Turing A.: Computing machinery and intelligence. Mind 59, 433–460 (1950)MathSciNetCrossRefGoogle Scholar
  157. van Barneveld, J., van Setten, M.: Involving users in the design of user interfaces for TV recommender systems. In: 3rd Workshop on Personalization in Future TV at UM03, Johnstown, PA (2003)Google Scholar
  158. van den Haak M.J., de Jong M.D.T., Schellens P.J.: Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav. Inf. Technol. 22(5), 339–351 (2003)CrossRefGoogle Scholar
  159. van den Haak M.J., de Jong M.D.T., Schellens P.J.: Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Interact. Comput. 16(6), 1153–1170 (2004)CrossRefGoogle Scholar
  160. van Velsen L., van der Geest T., Klaassen R., Steehouder M.: User-centered evaluation of adaptive and adaptable systems: a literature review. Knowl. Eng. Rev. 23(3), 261–281 (2008)Google Scholar
  161. VanLehn, K., Niu, Z., Siler, S., Gertner, A.S.: Student modeling from conventional test data: a Bayesian approach without priors. In: 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1452, pp. 434–443. Springer, Berlin (1998)Google Scholar
  162. Walker E., Rummel N., Koedinger K.R.: CTRL: A research framework for providing adaptive collaborative learning support. User Model. User-Adap. Inter. 19(5), 387–431 (2009)CrossRefGoogle Scholar
  163. Wang, Y., Chen, Z., Kobsa, A.: A collection and systematization of international privacy laws, with special consideration of internationally operating personalized websites. http://www.ics.uci.edu/~kobsa/privacy (2006)
  164. Weber, G., Specht, M.: User modeling and adaptive navigation support in WWW-based tutoring systems. In: 6th International Conference on User Modeling, Chia Laguna, Italy, pp. 289–300. Springer, Vienna (1997)Google Scholar
  165. Weibelzahl, S.: Evaluation of adaptive systems. In: 8th International Conference on User Modeling. LNCS, vol. 2109, pp. 292–294. Springer, Berlin (2001)Google Scholar
  166. Weibelzahl, S.: Evaluation of adaptive systems. PhD Thesis, University of Trier, Germany (2003)Google Scholar
  167. Weibelzahl, S.: Problems and pitfalls in evaluating adaptive systems. In: 4th Workshop on the Evaluation of Adaptive Systems at UM’05, pp. 57–66. Edinburgh, UK (2005)Google Scholar
  168. Weibelzahl, S., Weber, G.: Evaluating the inference mechanism of adaptive learning systems. In: 9th International Conference of User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 154–168. Springer, Berlin (2003)Google Scholar
  169. Wharton C., Rieman J., Lewis C., Polson P.: The cognitive walkthrough method: A practitioner’s guide. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 105–141. John Wiley & Sons, New York (1994)Google Scholar
  170. Wilson J., Rosenberg D.: Rapid prototyping for user interface design. In: Helander, M. (ed.) Handbook of Human–Computer Interaction, pp. 859–875. Elsevier, Amsterdam (1988)Google Scholar
  171. Winter, S., Wagner, S., Deissenboeck, F.: A comprehensive model of usability. In: Engineering Interactive Systems Conference. LNCS, vol. 4940, pp. 106–122. Springer, Berlin (2008)Google Scholar
  172. Witten I.A., Frank E.: Data Mining: Practical Machine Learning Tools and Techniques. 2nd edn. Morgan Kaufmann, Amsterdam (2005)MATHGoogle Scholar
  173. Yang, D., Huo, H.: Assessment on the adaptivity of adaptive systems. In: International Conference on Management of e-Commerce and e-Government, pp. 437-440. IEEE, Nanchang, China (2008)Google Scholar
  174. Yudelson M., Medvedeva O., Crowley R.: A multifactor approach to student model evaluation. User Model. User-Adap. Inter. 18(4), 349–382 (2008)CrossRefGoogle Scholar
  175. Zancanaro, M., Kuflik, T., Boger, Z., Goren-Bar, D., Goldwasser, D.: Analyzing museum visitors’ behavior patterns. In: 11th International Conference on User Modeling, Corfu, Greece. LNCS, vol. 4511, pp. 238–246. Springer, Berlin (2007)Google Scholar
  176. Zaslow, J.: If TiVo thinks you are gay, here’s how to set it straight. Wall Street J. A 1 (Nov 26, 2002)Google Scholar
  177. Zhang, T., Rau, P., Salvendy, G.: Developing instrument for handset usability evaluation: A survey study. In: 12th International Conference on Human–Computer Interaction, Beijing, China. LNCS, vol. 4550, pp. 662–671. Springer, Berlin (2007)Google Scholar
  178. Ziegler J., Bullinger J.H.: Formal models and techniques in Human–Computer interaction. In: Shackel, B., Richardson, S.J. (eds) Human Factors for Informatics Usability, pp. 183–206. Cambridge University Press, Cambridge (1991)Google Scholar
  179. Ziegler, C.-N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: 14 th International World Wide Web Conference, pp. 22–32. ACM, Chiba, Japan (2005)Google Scholar
  180. Zimmermann A., Lorenz A.: LISTEN: A user-adaptive audio-augmented museum guide. User Model. User-Adap. Inter. 18(5), 389–416 (2008)CrossRefGoogle Scholar
  181. Zimmermann A., Specht M., Lorenz A.: Personalization and context management. User Model. User-Adap. Inter. 15(3-4), 275–302 (2005)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • Alexandros Paramythis
    • 1
  • Stephan Weibelzahl
    • 2
  • Judith Masthoff
    • 3
  1. 1.Institute for Information Processing and Microprocessor Technology (FIM)Johannes Kepler University LinzLinzAustria
  2. 2.National College of Ireland DublinDublin 1Ireland
  3. 3.University of AberdeenAberdeenScotland, UK

Personalised recommendations