Skip to main content

A User Modeling Shared Challenge Proposal

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11696))

  • 1113 Accesses

Abstract

Comparative evaluation in the areas of User Modeling, Adaptation and Personalization (UMAP) is significantly challenging. It has always been difficult to rigorously compare different approaches to personalization, as the function of the resulting systems is, by their nature, heavily influenced by the behavior of the users involved in trialing the systems. Developing comparative evaluations in this space would be a huge advancement as it would enable shared comparison across research. Here we present a proposal for a shared challenge generation in UMAP, focusing on user model generation using logged mobile phone data, with an assumed purpose of supporting mobile phone notification suggestion. The dataset, evaluation metrics, and challenge operation are described.

Authors listed alphabetically.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://evalumap.adaptcentre.ie.

  2. 2.

    http://www.ir.disco.unimib.it/pir-clef2019/.

  3. 3.

    https://www.acm.org/publications/policies/artifact-review-badging.

References

  1. Conlan, O., Kelly, L., Koidl, K., Lawless, S., Levacher, K., Staikopoulos, A.: EvalUMAP 2016: towards comparative evaluation in the user modelling, adaptation and personalization space workshop. In: UMAP 2016 (2016)

    Google Scholar 

  2. Conlan, O., Kelly, L., Koidl, K., Lawless, S., Staikopoulos, A.: EvalUMAP 2017: towards comparative evaluation in the user modelling, adaptation and personalization space workshop. In: UMAP 2017 (2017)

    Google Scholar 

  3. Fehr, J., Heiland, J., Himpe, C., Saak, J.: Best practices for replicability, reproducibility and reusability of computer-based experiments exemplified by model reduction software. arXiv preprint arXiv:1607.01191 (2016)

  4. Fraser, K., Yousuf, B., Conlan, O.: Synthesis and evaluation of a mobile notification dataset. In: Adjunct Publication of UMAP 2017 (2017)

    Google Scholar 

  5. Fraser, K., Yousuf, B., Conlan, O.: Scrutable and persuasive push-notifications. In: Oinas-Kukkonen, H., Win, K.T., Karapanos, E., Karppinen, P., Kyza, E. (eds.) PERSUASIVE 2019. LNCS, vol. 11433, pp. 67–73. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17287-9_6

    Chapter  Google Scholar 

  6. Hopfgartner, F., Kille, B., Lommatzsch, A., Plumbaum, T., Brodt, T., Heintz, T.: Benchmarking news recommendations in a living lab. In: Kanoulas, E., et al. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 250–267. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11382-1_21

    Chapter  Google Scholar 

  7. Hui Yang, G., Soboroff, I.: TREC 2016 dynamic domain track overview. In: TREC 2016 (2016)

    Google Scholar 

  8. Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User-Adap. Interact. 20(5), 383–453 (2010). https://doi.org/10.1007/s11257-010-9082-4

    Article  Google Scholar 

  9. Park, K.S., Hwan Lim, C.: A structured methodology for comparative evaluation of user interface designs using usability criteria and measures. Int. J. Ind. Ergon. 23(5–6), 379–389 (1999). https://doi.org/10.1016/S0169-8141(97)00059-0

    Article  Google Scholar 

  10. Schuth, A., Balog, K., Kelly, L.: Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF lab 2015. In: Mothe, J., et al. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 484–496. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24027-5_47

    Chapter  Google Scholar 

  11. Van Velsen, L., van der Geest, T., Klaassen, R., Steehouder, M.: User-centered evaluation of adaptive and adaptable systems: a literature review. Knowl. Eng. Rev. 23(3), 261–281 (2008). https://doi.org/10.1017/S0269888908001379

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Liadh Kelly .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Conlan, O., Fraser, K., Kelly, L., Yousuf, B. (2019). A User Modeling Shared Challenge Proposal. In: Crestani, F., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2019. Lecture Notes in Computer Science(), vol 11696. Springer, Cham. https://doi.org/10.1007/978-3-030-28577-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-28577-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-28576-0

  • Online ISBN: 978-3-030-28577-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics