Skip to main content
  • Book
  • Open Access
  • © 2021

Evaluating Information Retrieval and Access Tasks

NTCIR's Legacy of Research Impact

  • Describes the creation of shared-task evaluations for information access technology

  • Examines a dozen research directions pioneered by NTCIR over two decades

  • Highlights the practical impact of NTCIR contributions to the information industry

Part of the book series: The Information Retrieval Series (INRE, volume 43)

Buying options

Softcover Book USD 49.99
Price excludes VAT (USA)
Hardcover Book USD 59.99
Price excludes VAT (USA)

Table of contents (14 chapters)

  1. Front Matter

    Pages i-xiii
  2. Graded Relevance

    • Tetsuya Sakai
    Pages 1-20Open Access
  3. Text Summarization Challenge: An Evaluation Program for Text Summarization

    • Hidetsugu Nanba, Tsutomu Hirao, Takahiro Fukushima, Manabu Okumura
    Pages 39-48Open Access
  4. Challenges in Patent Information Retrieval

    • Makoto Iwayama, Atsushi Fujii, Hidetsugu Nanba
    Pages 49-69Open Access
  5. Multi-modal Summarization

    • Tsuneaki Kato
    Pages 71-82Open Access
  6. Opinion Analysis Corpora Across Languages

    • Yohei Seki
    Pages 83-95Open Access
  7. Patent Translation

    • Isao Goto
    Pages 97-107Open Access
  8. Component-Based Evaluation for Question Answering

    • Teruko Mitamura, Eric Nyberg
    Pages 109-126Open Access
  9. Temporal Information Access

    • Masaharu Yoshioka, Hideo Joho
    Pages 127-141Open Access
  10. SogouQ: The First Large-Scale Test Collection with Click Streams Used in a Shared-Task Evaluation

    • Ruihua Song, Min Zhang, Cheng Luo, Tetsuya Sakai, Yiqun Liu, Zhicheng Dou
    Pages 143-150Open Access
  11. Evaluation of Information Access with Smartphones

    • Makoto P. Kato
    Pages 151-167Open Access
  12. Mathematical Information Retrieval

    • Akiko Aizawa, Michael Kohlhase
    Pages 169-185Open Access
  13. Experiments in Lifelog Organisation and Retrieval at NTCIR

    • Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, Rami Albatal, Graham Healy et al.
    Pages 187-203Open Access
  14. The Future of Information Retrieval Evaluation

    • Douglas W. Oard
    Pages 205-216Open Access
  15. Back Matter

    Pages 217-219

About this book

This open access book summarizes the first two decades of the NII Testbeds and Community for Information access Research (NTCIR). NTCIR is a series of evaluation forums run by a global team of researchers and hosted by the National Institute of Informatics (NII), Japan. The book is unique in that it discusses not just what was done at NTCIR, but also how it was done and the impact it has achieved. For example, in some chapters the reader sees the early seeds of what eventually grew to be the search engines that provide access to content on the World Wide Web, today’s smartphones that can tailor what they show to the needs of their owners, and the smart speakers that enrich our lives at home and on the move. We also get glimpses into how new search engines can be built for mathematical formulae, or for the digital record of a lived human life.

Key to the success of the NTCIR endeavor was early recognition that information access research is an empirical discipline and that evaluation therefore lay at the core of the enterprise. Evaluation is thus at the heart of each chapter in this book. They show, for example, how the recognition that some documents are more important than others has shaped thinking about evaluation design. The thirty-three contributors to this volume speak for the many hundreds of researchers from dozens of countries around the world who together shaped NTCIR as organizers and participants.

This book is suitable for researchers, practitioners, and students—anyone who wants to learn about past and present evaluation efforts in information retrieval, information access, and natural language processing, as well as those who want to participate in an evaluation task or even to design and organize one.


  • Evaluation
  • Information Retrieval
  • Multilingual Information Access
  • Test Collections
  • Information Search
  • Information Storage
  • Artificial Intelligence
  • Open Acces

Editors and Affiliations

  • Department of Computer Science and Engineering, Waseda University, Tokyo, Japan

    Tetsuya Sakai

  • College of Information Studies, University of Maryland, College Park, USA

    Douglas W. Oard

  • Information-Society Research Division, National Institute of Informatics, Tokyo, Japan

    Noriko Kando

About the editors

Tetsuya Sakai is a professor in the Department of Computer Science and Engineering, Waseda University, Japan. He is also a visiting professor at the National Institute of Informatics. He joined Toshiba in 1993 and obtained a Ph.D. from Waseda in 2000. In 2000 and 2001, he was supervised by the late Karen Sparck Jones at the Computer Laboratory, University of Cambridge, as a visiting researcher. In 2007, he joined NewsWatch, Inc. In 2009, he joined Microsoft Research Asia and the Waseda faculty in 2013. He was an associate dean (IT Strategies Division) there from 2015 to 2017, and department head from 2017 to 2019. He is the editor-in-chief of the Information Retrieval Journal (Springer) and an associate editor of ACM Transactions on Information Systems (TOIS). He is a distinguished member of the ACM, and the current vice chair of the ACM SIGIR (July 2019 – June 2022). 

Douglas W. Oard is a professor in the College of Information Studies (Maryland’s iSchool) and the University of Maryland Institute for Advanced Computer Studies (UMIACS). He holds affiliate appointments in Maryland’s Computer Science Department and Maryland’s Applied Mathematics, Statistics, and Scientific Computing program. He earned his Ph.D. in electrical engineering from the University of Maryland, College Park, and his B.A. and M.E.E. degrees in electrical engineering from Rice University. He is the current associate dean for faculty there. Previously, he was the associate dean for research in the College of Information Studies, and the director of the UMIACS Computational Linguistics and Information Processing (CLIP) laboratory.

Noriko Kando is a professor in the Information-Society Research Division of the National Institute of Informatics (NII), Tokyo, Japan, and also has been appointed as a professor in the Department of Informatics at the Graduate University of Advanced Studies (SOKENDAI), Japan. She initiated NTCIR in late 1997, an evaluation of information-access technologies such as information retrieval, summarization, question answering, and text mining, using various types of documents in East Asian languages and English, and has been a designer of various tasks and general co-chair of NTCIR. She received her Ph.D. from Keio University in library and information science in 1995 and has been teaching and conducting research at NII since 1994.

Bibliographic Information

  • Book Title: Evaluating Information Retrieval and Access Tasks

  • Book Subtitle: NTCIR's Legacy of Research Impact

  • Editors: Tetsuya Sakai, Douglas W. Oard, Noriko Kando

  • Series Title: The Information Retrieval Series

  • DOI:

  • Publisher: Springer Singapore

  • eBook Packages: Computer Science, Computer Science (R0)

  • Copyright Information: The Editor(s) (if applicable) and The Author(s) 2021

  • License: CC BY

  • Hardcover ISBN: 978-981-15-5553-4

  • Softcover ISBN: 978-981-15-5556-5

  • eBook ISBN: 978-981-15-5554-1

  • Series ISSN: 1871-7500

  • Series E-ISSN: 2730-6836

  • Edition Number: 1

  • Number of Pages: XIII, 219

  • Number of Illustrations: 14 b/w illustrations, 11 illustrations in colour

  • Topics: Information Storage and Retrieval

Buying options

Softcover Book USD 49.99
Price excludes VAT (USA)
Hardcover Book USD 59.99
Price excludes VAT (USA)