Evaluating Personal Information Retrieval

Abstract

Evaluation of personal search over an individual’s personal information space on the desktop or elsewhere is problematic for reasons relating both to the personal and private nature of the data and the associated personal information needs of collection owners. Indeed challenges associated with evaluation in this space are recognised as one of the key factors hindering the development of research in personal information retrieval. We present the “personal information retrieval evaluation (PIRE)” tool, which provides a solution to this evaluation problem using a ‘living laboratory’ approach. This tool allows for the evaluation of retrieval techniques using ‘real’ individuals’ personal collections, queries and result sets, in a cross-comparable repeatable way, while importantly maintaining an individual’s informational privacy.