Information Retrieval

, Volume 17, Issue 4, pp 351–379 | Cite as

Evaluating hierarchical organisation structures for exploring digital libraries

  • Mark M. HallEmail author
  • Samuel Fernando
  • Paul D. Clough
  • Aitor Soroa
  • Eneko Agirre
  • Mark Stevenson


Search boxes providing simple keyword-based search are insufficient when users have complex information needs or are unfamiliar with a collection, for example in large digital libraries. Browsing hierarchies can support these richer interactions, but many collections do not have a suitable hierarchy available. In this paper we present a number of approaches for automatically creating hierarchies and mapping items into them, including a novel technique which automatically adapts a Wikipedia-based taxonomy to the target collection. These approaches are applied to a large collection of cultural heritage items which is formed through the aggregation of other collections and for which no unified hierarchy is available. We investigate a number of novel user-evaluated metrics to quantify the hierarchies’ quality and performance, showing that the proposed technique is preferred by users. From this we draw a number of conclusions as to what makes a hierarchy useful to the user.


Evaluation Hierarchical structures Exploratory search Interactive information retrieval Browsing 



The research leading to these results was supported by the PATHS project ( funded by the European Community’s Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No. 270082.


  1. Anick, P., & Tipirneni, S. (1999) The paraphrase search assistant: Terminological feedback for iterative information seeking. In Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval, ACM (pp. 153–159).Google Scholar
  2. Atserias, J., Villarejo, L., Rigau, G., Agirre, E., Carroll, J., Magnini, B. et al. (2004). The meaning multilingual central repository. In Proceedings of GWC (pp. 23–30).Google Scholar
  3. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., & Ives, Z. (2007). Dbpedia: A nucleus for a web of open data. In The Semantic Web (pp. 722–735).Google Scholar
  4. Azzopardi, L., Girolami, M., & Van Rijsbergen, C. (2004). Topic based language models for ad hoc information retrieval. In Neural networks, 2004. Proceedings. 2004 IEEE international joint conference on, IEEE (Vol. 4, pp. 3281–3286).Google Scholar
  5. Blei, D. M., Griffiths, T., Jordan, M., & Tenenbaum, J. (2003). Hierarchical topic models and the nested chinese restaurant process. In NIPS.
  6. Borlund, P., & Ingwersen, P. (1997). The development of a method for the evaluation of interactive information retrieval systems. Journal of documentation, 53(3), 225–250.CrossRefGoogle Scholar
  7. Brewster, C., Alani, H., Dasmahapatra, S., & Wilks, Y. (2004). Data driven ontology evaluation. In Proceedings of international conference on language resources and evaluation.Google Scholar
  8. Carterette, B., Bennett, P. N., Chickering, D. M., & Dumais, S. T. (2008). Here or there: Preference judgements for relevance. In C. Macdonald, I. Ounis, V. Plachouras, I. Ruthven, & R. W. White (Eds.), Proceedings of the IR research, 30th European conference on advances in information retrieval (ECIR’08) (pp. 16–27). Berlin, Heidelberg: Springer.Google Scholar
  9. Chang, J., Boyd-Graber, J., Wang, C., Gerrish, S., & Blei, D. M. (2009). Reading tea leaves: How humans interpret topic models. In NIPS.Google Scholar
  10. Chen, M., Hearst, M., Hong, J., & Lin, J. (1999). Cha-cha: A system for organizing intranet search results. In Proceedings of the 2nd conference on USENIX symposium on internet technologies and systems (pp. 11–14).Google Scholar
  11. Falleti, M. G., Maruff, P., Collie, A., & Darby, D. G. (2006). Practice effects associated with the repeated assessment of cognitive function using the cogstate battery at 10-minute, one week and one month test–retest intervals. Journal of Clinical and Experimental Neuropsychology, 28(7), 1095–1112.CrossRefGoogle Scholar
  12. Fellbaum, C. (1998). WordNet: An electronic database. Cambridge, MA: MIT Press.zbMATHGoogle Scholar
  13. Fernando, S., Hall, M., Agirre, E., Soroa, A., Clough, P., & Stevenson, M. (2012). Comparing taxonomies for organising collections of documents. In Proceedings of COLING 2012, The COLING 2012 Organizing Committee, Mumbai, India (pp. 879–894).
  14. Gómez-Pérez, A. (1996). Towards a framework to verify knowledge sharing technology. Expert Systems with Applications, 11(4), 519–529.CrossRefGoogle Scholar
  15. Hall, M. M., & Toms, E. (2013). Building a common framework for iir evaluation. In CLEF 2013—Information access evaluation. Multilinguality, multimodality, and visualization (pp. 17–28). doi: 10.1007/978-3-642-40802-1_3.
  16. Hearst, M. (2006a). Clustering versus faceted categories for information exploration. Communications of the ACM, 49(4), 59–61.CrossRefGoogle Scholar
  17. Hearst, M. (2006b). Design recommendations for hierarchical faceted search interfaces. In Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR’06) workshop on faceted search.Google Scholar
  18. Hoffart, J., Suchanek, F., Berberich, K., Lewis-Kelham, E., De Melo, G., & Weikum, G. (2011). Yago2: Exploring and querying world knowledge in time, space, context, and many languages. In Proceedings of the 20th international conference companion on World Wide Web, ACM (pp. 229–232).Google Scholar
  19. Hornbæk, K., & Hertzum, M. (2011). The notion of overview in information visualization. International Journal of Human-Computer Studies, 69(7–8), 509–525. doi:  10.1016/j.ijhcs.2011.02.007.
  20. Horvat, M., Grbin, A., & Gledec, G. (2012) Wntags: A web-based tool for image labeling and retrieval with lexical ontologies. In: M. Graña, C. Toro, J. Posada, R. J. Howlett & L. C. Jain (Eds.), Frontiers in artificial intelligence and applications (Vol. 243, pp. 585–594). KES, IOS Press.Google Scholar
  21. Jörgensen, C. (2004). Unlocking the museum: A manifesto. Journal of the American Society for Information Science and Technology, 55(5), 462–464. doi: 10.1002/asi.10396.CrossRefGoogle Scholar
  22. Kelly, D., & Sugimoto, C. (2013). A systematic review of interactive information retrieval evaluation studies, 1967–2006. JASIST, 64(4), 745–770.CrossRefGoogle Scholar
  23. Lau, J., Grieser, K., Newman, D., & Baldwin, T. (2011). Automatic labelling of topic models. In Proceedings of the 49th annual meeting on association for computational linguistics (pp. 1536–1545).Google Scholar
  24. Lawrie, D., Croft, W., & Rosenberg, A. (2001). Finding topic words for hierarchical summarization. In Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, ACM (pp. 349–357).Google Scholar
  25. Liu, X., Song, Y., Liu, S., & Wang, H. (2012). Automatic taxonomy construction from keywords. In Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining, ACM, New York, NY, USA, KDD ’12 (pp. 1433–1441). doi: 10.1145/2339530.2339754.
  26. Maedche, A., & Staab, S. (2002). Measuring similarity between ontologies. In Knowledge engineering and knowledge management: Ontologies and the semantic web (pp. 15–21).Google Scholar
  27. Magnini, B., & Cavaglia, G. (2000). Integrating subject field codes into wordnet. In Proceedings of LREC-2000, second international conference on language resources and evaluation (pp. 1413–1418).Google Scholar
  28. Marchionini, G. (2006). Exploratory search: From finding to understanding. Communications of the ACM, 49(4), 41–46.CrossRefGoogle Scholar
  29. Markkula, M., & Sormunen, E. (2000). End-user searching challenges indexing practices in the digital newspaper photo archive. Information Retrieval, 1(4), 259–285.CrossRefzbMATHGoogle Scholar
  30. Milne, D., & Witten, I. H. (2008). Learning to link with Wikipedia. In Proceedings of the 17th ACM conference on information and knowledge management (pp. 509–518).Google Scholar
  31. Milne, D. N., Witten, I. H., & Nichols, D. M. (2007). A knowledge-based search engine powered by Wikipedia. In Proceedings of the sixteenth ACM conference on conference on information and knowledge management, ACM (pp. 445–454).Google Scholar
  32. Navigli, R., Velardi, P., & Gangemi, A. (2003). Ontology learning and its application to automated terminology translation. Intelligent Systems, IEEE, 18(1), 22–31.CrossRefGoogle Scholar
  33. Nevill-Manning, C., Witten, I., & Paynter, G. (1999). Lexically-generated subject hierarchies for browsing large collections. International Journal on Digital Libraries, 2(2), 111–123.CrossRefGoogle Scholar
  34. Padró, L., Reese, S., Agirre, E., & Soroa, A. (2010). Semantic services in freeling 2.1: Wordnet and ukb. In P. Bhattacharyya, C. Fellbaum, & P. Vossen (Eds.), Principles, construction, and application of multilingual Wordnets, global Wordnet conference 2010 (pp. 99–105). Mumbai, India: Narosa Publishing House.Google Scholar
  35. Pirolli, P. (2009). Powers of 10: Modeling complex information-seeking systems at multiple scales. Computer, 42(3), 33–40. doi: 10.1109/MC.2009.94.CrossRefGoogle Scholar
  36. Pirolli, P., Schank, P., Hearst, M., & Diehl, C. (1996). Scatter/gather browsing communicates the topic structure of a very large text collection. In Proceedings of the SIGCHI conference on human factors in computing systems: common ground, ACM (pp. 213–220).Google Scholar
  37. Ponzetto, S., & Strube, M. (2011). Taxonomy induction based on a collaboratively built knowledge repository. Artificial Intelligence, 175(9–10), 1737–1756.CrossRefMathSciNetGoogle Scholar
  38. Pratt, W., Hearst, M., & Fagan, L. (1999). A knowledge-based approach to organizing retrieved documents. In Proceedings of th 16th annual conference on artificial intelligence (AAAI 99).Google Scholar
  39. Rao, R., Pedersen, J. O., Hearst, M. A., Mackinlay, J. D., Card, S. K., Masinter, L., et al. (1995). Rich interaction in the digital library. Communications of the ACM, 38(4), 29–39. doi: 10.1145/205323.205326.CrossRefGoogle Scholar
  40. Rosenfeld, L., & Morville, P. (2002). Information architecture for the World Wide Web: Designing large-scale web sites. Sebastopol: O’Reilly Media, Incorporated.Google Scholar
  41. Sanderson, M., & Croft, B. (1999). Deriving concept hierarchies from text. In Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval, ACM (pp. 206–213).Google Scholar
  42. Shiri, A., Revie, C., & Chowdhury, G. (2002). Thesaurus-enhanced search interfaces. Journal of Information Science, 28(2), 111–122.CrossRefGoogle Scholar
  43. Singer, G., Norbisrath, U., & Lewandowski, D. (2012). Ordinary search engine users carrying out complex search tasks. Journal of Information Science, 39(3), 346–358.Google Scholar
  44. Skov, M., & Ingwersen, P. (2008). Exploring information seeking behaviour in a digital museum context. In Proceedings of the second international symposium on Information interaction in context, ACM (pp. 110–115).Google Scholar
  45. Stoica, E., Hearst, M., & Richardson, M. (2007). Automating creation of hierarchical faceted metadata structures. In Human language technologies: The annual conference of the North American chapter of the association for computational linguistics (NAACL-HLT 2007) (pp. 244–251).Google Scholar
  46. Tang, L., Zhang, J., & Liu, H. (2006). Acclimatizing taxonomic semantics for hierarchical content classification. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining, ACM, New York, NY, USA, KDD ’06 (pp. 384–393). doi: 10.1145/1150402.1150446.
  47. Toms, E. G., Villa, R., & McCay-Peet, L. (2013). How is a search system used in work task completion? Journal of Information Science, 39(1), 15–25.CrossRefGoogle Scholar
  48. Treeratpituk, P., & Callan, J. (2006). Automatically labeling hierarchical clusters. In Proceedings of the 2006 international conference on Digital Government Research, Digital Government Society of North America, dg.o ’06 (pp. 167–176). doi: 10.1145/1146598.1146650.
  49. Wang, Z., Khoo, C. S., & Chaudhry, A. S. (2014). Evaluation of the navigation effectiveness of an organizational taxonomy built on a general classification scheme and domain thesauri. Journal of the Association for Information Science and Technology,. doi: 10.1002/asi.23017.Google Scholar
  50. Wei, X., & Croft, W. (2006) LDA-based document models for ad-hoc retrieval. In Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, ACM (pp. 178–185).Google Scholar
  51. White, R. W., Kules, B., Drucker, S. M., & Schraefel, M. (2006). Introduction. Communications of the ACM, 49(4), 36–39. doi: 10.1145/1121949.1121978.CrossRefGoogle Scholar
  52. Yakel, E., Shaw, S., & Reynolds, P. (2007). Creating the next generation of archival finding aids. D-Lib Magazine, 13(5/6). doi: 10.1045/may2007-yakel.
  53. Yu, J., Thom, J., & Tam, A. (2007). Ontology evaluation using Wikipedia categories for browsing. In Proceedings of the sixteenth ACM conference on conference on information and knowledge management, ACM (pp. 223–232).Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Mark M. Hall
    • 1
    Email author
  • Samuel Fernando
    • 2
  • Paul D. Clough
    • 3
  • Aitor Soroa
    • 4
  • Eneko Agirre
    • 4
  • Mark Stevenson
    • 2
  1. 1.Department of ComputingEdge Hill UniversityOrmskirkUK
  2. 2.Department of Computer ScienceSheffield UniversitySheffieldUK
  3. 3.Information SchoolSheffield UniversitySheffieldUK
  4. 4.IXA NLP GroupUniversity of the Basque CountryDonostiaSpain

Personalised recommendations