Evaluating the Adaptation of a Learning System before the Prototype Is Ready: A Paper-Based Lab Study

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5535)


We report on results of a paper-based lab study that used information on task performance, self appraisal and personal learning need assessment to validate the adaptation mechanisms for a work-integrated learning system. We discuss the results in the wider context of the evaluation of adaptive systems where the validation methods we used can be transferred to a work-based setting to iteratively refine adaptation mechanisms and improve model validity.


Adaptive Learning Systems Evaluation Task-based Competency Assessment Learning Need Analysis Knowledge Space Theory 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Van Velsen, L., Van Der Geest, T., Klaassen, R., Steehouder, M.: User-centered evaluation of adaptive and adaptable systems: a literature review. The Knowledge Engineering Review 23(3), 261–281 (2008)Google Scholar
  2. 2.
    Brusilovsky, P., Karagiannidis, C., Sampson, D.: The Benefits of Layered Evaluation of Adaptive Applications and Services. In: Weibelzahl, S., Chin, D., Weber, G. (eds.) Empirical evaluation of adaptive systems. Workshop at the UM 2001, pp. 1–8 (2001)Google Scholar
  3. 3.
    Paramythis, A., Totter, A., Stephanidis, C.: A modular approach to the evaluation of adaptive user interfaces. In: Weibelzahl, S., Chin, D., Weber, G. (eds.) Empirical evaluation of adaptive systems: Workshop at the UM 2001, pp. 9–24 (2001) Google Scholar
  4. 4.
    Weibelzahl, S., Lauer, C.U.: Framework for the evaluation of adaptive CBR-systems. In: Vollrath, I., Schmitt, S., Reimer, U. (eds.) Experience Management as Reuse of Knowledge. GWCBR 2001, pp. 254–263. Baden-Baden, Germany (2001)Google Scholar
  5. 5.
    Lindstaedt, S.N., Ley, T., Scheir, P., Ulbrich, A.: Applying Scruffy Methods to Enable Work-integrated Learning. Upgrade: The European Journal of the Informatics Professional 9(3), 44–50 (2008)Google Scholar
  6. 6.
    Korossy, K.: Extending the theory of knowledge spaces: A competence-performance approach. Zeitschrift für Psychologie 205, 53–82 (1997)Google Scholar
  7. 7.
    Ley, T., Ulbrich, A., Scheir, P., Lindstaedt, S.N., Kump, B., Albert, D.: Modelling Competencies for supporting Work-integrated Learning in Knowledge Work. Journal of Knowledge Management 12(6), 31–47 (2008)CrossRefGoogle Scholar
  8. 8.
    Maiden, N.A., Jones, S.V.: The RESCUE Requirements Engineering Process - An Integrated User-centered Requirements Engineering Process, Version 4.1. Centre for HCI Design, The City University, London/UK (2004)Google Scholar
  9. 9.
    Hoffman, C., Nathan, B., Holden, L.: A Comparison of Validation Criteria: Objective versus Subjective Performance Measures and Self- versus Supervisor Ratings. Personnel Psychology 44, 601–619 (1991)CrossRefGoogle Scholar
  10. 10.
    Mabe, P., West, S.: Validity of Self-Evaluation of Ability: A Review and Meta-Analysis. Journal of Applied Psychology 67, 280–296 (1982)CrossRefGoogle Scholar
  11. 11.
    Ley, T.: Organizational Competency Management - A Competence Performance Approach. Shaker, Aachen (2006)Google Scholar
  12. 12.
    Muellerbuchhof, R., Zehrt, P.: Vergleich subjektiver und objektiver Messverfahren für die Bestimmung von Methodenkompetenz am Beispiel der Kompetenzmessung bei technischem Fachpersonal. Zeitschrift für Arbeits- und Organisationspsychologie 48, 132–138 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  1. 1.Know-CenterGrazAustria
  2. 2.Cognitive Science SectionUniversity of GrazGrazAustria
  3. 3.Knowledge Management InstituteGraz University of TechnologyGrazAustria
  4. 4.Centre for HCI DesignCity University LondonLondonUnited Kingdom

Personalised recommendations