Abstract
In the IR field it is clear that the value of a system depends on the cost and benefit profiles of its users. It would seem obvious that different users would prefer different systems. In the TREC-9 filtering track, systems are evaluated by a utility measure specifying a given cost and benefit. However, in the study of decision systems it is known that, in some cases, one system may be unconditionally better than another. In this paper we employ a decision theoretic approach to find conditions under which an Information Filtering (IF) system is unconditionally superior to another for all users regardless of their cost and benefit profiles.
It is well known that if two IF systems have equal precision the system with better recall will be preferred by all users. Similarly, with equal recall, better precision is universally preferred. We confirm these known results and discover an unexpected dominance relation in which a system with lower recall will be universally preferred provided its precision is sufficiently higher.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Ahituv NA (1981) Comparison of information structures for a “rigid decision rule” case. Decision Sciences, 12:399-416.
Ahituv N and Ronen B (1988) Orthogonal information structures-A model to evaluate the information provided by second opinion. Decision Sciences, 19:255-268.
Ahituv N and Wand Y (1984) Comparative estimation of information under two business objectives. Decision Sciences, 15:31-51.
Baeza-Yates R and Ribeiro-Neto B (1999) Modern Information Retrieval. ACM Press, Addison-Wesley, New York.
Belkin NJ and Croft WB (1992) Information filtering and information retrieval two sides of the same coin? Communications of the ACM, 35(12):29-38.
Blackwell D (1951) Comparison of experiments. In: Neyman J, Ed., Proceedings of the Second Berekeley Symposium on Mathematical Statistics and Probability, pp. 93-102.
Blackwell D and Girshick MA (1954, 1979) Theory of Games and Statistical Decisions. John Wiley and Sons (1954), Dover (1979), p. 353.
Carmi N and Ronen B (1996), An information value approach to quality control attribute sampling. European Journal of Operational Research, 92(3):615-627.
Demski JS (1972) Information Analysis. Addison-Wesley, Reading, MA.
Frakes WB and Baeza-Yates R (1992) Information Retrieval: Data Structures and Algorithms. Prentice-Hall, Englewood Cliffs, NJ.
Hanani U, Shapira B and Shoval P (2001) Information filtering: Overview of issues, research and systems. User Modeling and User-Adapted Interaction, 11:203-259.
Hilton RW (1990) Failure of Blackwell's Theorem under Machina's generalization of expected utility analysis without the independence axiom. Journal of Economics Behavior & Organization, 13(2):233-244.
Hull DA and Robertson S (2000) The TREC-8 filtering track final report. In: Voorhees EM and Harman DKm, Eds., The 8th Text Retrieval Conference (TREC-8), NIST SP 500-246, pp. 35-36.
Kantor PB and Voorhees EM (2000) The TREC-5 confusion track: Comparing retrieval methods for scanned text. Information Retrieval, v2:165-177.
Luce RD and Winterfeldt D (1994) What common ground exists for descriptive, prescriptive, and normative utility theories, Management Science, 40:263-279.
Marschak J (1971) Economics of information systems. Journal of the American Statistical Association, 192-219.
Marshall RM and Narashimhan R (1989) Risk constrained information choice. Decision Sciences, 20(4):677-684.
McGuire CB and Radner R (1986), Eds. Decision and Organization. North Holland, Amsterdam.
Oard WD (1997) The state of the art in text filtering. User Modeling and User Adapted Interaction (UMUAI), 7(3):141-178.
Robertson SE (2002) Overview of The TREC routing and filtering tasks. Information Retrieval, 5:127-137
Ronen B (1994) An information value approach to quality control attribute sampling, European Journal of Operational Research, 73:430-442.
Ronen B and Spector Y (1995) Evaluating sampling strategy under two criteria. European Journal of Operational Research, 80:59-67.
Sinchcombe MB (1990) Baysian information topologies. Journal of Mathematical Economics, 19(3):233-253.
Tauge-Sutcliffe J (1992) Measuring the informativeness of a retrieval process. In: Proceedings of the Fifteen Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 23-36.
Van Rijsbergen C (1979) Information Retrieval, 2nd edn. Butterworth, London (Especially chapter 7).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Elovici, Y., Shapira, B. & Kantor, P.B. Using the Information Structure Model to Compare Profile-Based Information Filtering Systems. Information Retrieval 6, 75–97 (2003). https://doi.org/10.1023/A:1022952531694
Issue Date:
DOI: https://doi.org/10.1023/A:1022952531694