This chapter discusses our work in evaluating question-answering systems. We begin by reviewing evaluation concepts and methods that have been used in the past. We then consider more recent efforts to evaluate search engines that have contributed to the techniques we currently use. Finally we discuss our techniques and discuss results from user studies we have conducted. Implications are discussed for the future design of habitable question-answering systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
6. References
Bates, Marcia J. “An Exploratory Paradigm for Online Information Retrieval.” Intelligent Information Systems for the Information Society. Ed. B.C. Brookes. Amsterdam: North-Holland, 1986. 91-99.
Bernick, Philip. “Habitability in Search Engine Interfaces: Characteristics Identified Through Formative Evaluation Techniques.” Doctoral dissertation, New Mexico State University, Las Cruces, NM, 2003.
Catledge, Lara D., and James E. Pitkow. “Characterizing Browsing Strategies in the World-Wide Web.” Computer Networks and ISDN Systems. 27.6 (1995): 1065-73.
Choo, C.W., B. Detlor, and D. Turnbull. “Information Seeking on the Web-An integrated model of browsing and searching.” Proceedings of the Annual Meeting of the American Society for Information Science (ASIS), 23 June 2003 <http://choo.fis.utoronto.ca/fis/respub/aisis99/>
Chu, H., and Rosenthal, M. 1996. Search Engines for the World Wide Web: A Comparative Study and Evaluation Methodology. In ASIS1996 Annual Conference Proceedings,23 June2003 http://www.asis.org/annual-96/ ElectronicProceedings/chu.html
Cockburn, A., & Jones, S. (1996). Which Way Now? Analysing and Easing Inadequacies in WWW Navigation. International Journal of Human-Computer Studies, 45(1), pp. 105-129.
Hölscher, Christoph and Gerhard Strube. “Web Search Behavior of Internet Experts and Newbies.” Proceedings of the9th International World Wide Web Conference(WWW9). 23 June 2003 http://www.www9.org/w9cdrom/81/81.html
Jansen, B.J., A. Spink, J. Bateman, and T. Saracevic. “Real Life Information Retrieval: A Study of User Queries on the Web.” SIGIR Forum, 32.1 (1998): 5-17.
Jansen, B.J., A. Spink, and T. Saracevic. “Failure Analysis in Query Constructions: Data and Analysis from a Large Sample of Web Queries.” Digital Libraries 98. New York: ACM, 1998. 289-90.
Katz, Boris, Sue Felshin, Deniz Yuret, Ali Ibrahim, Jimmy Lin, Gregory Marton, Alton Jerome McFarland and Baris Temelkuran. “Omnibase: Uniform Access to Heterogeneous Data for Question Answering.” Proceedings of the 7th International Workshop on Applications of Natural Language to Information Systems (NLDB 2002), June, 2002.
Lewis, Clayton and John Rieman. “Task-Centered User Interface Design: A Practical Introduction.” available via anonymous ftp at: ftp.cs.colorado.edu in:/pub/cs/distribs/clewis/HCI-Design-Book. 1994.
Marchionini, G., S. Dwiggins, A. Katz, and X. Lin. “Information Seeking in Full-Text End-User-Oriented Search Systems: The Roles of Domain and Search Expertise.” LISR, 15 (1993): 35-69.
Moldovan, Dan, Marius Pasca, Sanda Harabagiu, and Mihai Surdeanu. Performance Issues and Error Analysis in an Open-Domain Question Answering System. 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), July 2002, Philadelphia, Pennsylvania.
Navarro-Prieto, R., M. Scaife, and Y. Rogers. “Cognitive Strategies in Web Searching.” Proceedings of the 5th Conference on Human Factors and the Web, 23 June 2003 zing.ncsl.nist.gov/hfweb/ proceedings/navarro-prieto/index.html
O’day, Vicki, and Robin Jeffries. “Orienteering in an Information Landscape: How Information Seekers Get From Here to There.” HPL-92-127 Hewlett-Packard:n.p, 1992.
Ogden, William C., and Philp Bernick. “Oleada: User-Centered Tipster Technology for Language Instruction.” Proceedings of the Tipster Phase II 24 Month Workshop. Tysons Corner, VA, 1996.
Ogden, William C., and Philp Bernick. “Tabula Rasa Meta-Tool: Text Extraction Toolbuilder Toolkit.” Technical Report MCCS-97-305, Computing Research Laboratory, New Mexico State University, Las Cruces, NM, 1997.
Ogden, William. C., and Philip Bernick. “Using Natural Language Interfaces.” Handbook of Human Computer Interaction. Eds. Helender, M., T. Landauer, and P. Prabhu, 2nd Edition, Amsterdam: North Holland, 1997, 137-61.
Spink, A., D. Wolfram, B.J Jansen, and T. Saracevic. “Searching the Web: The Public and Their Queries.” Journal of the American Society for Information Science and Technology, 52 (2001): 226-34.
Tauscher, Linda and Saul Greenberg. “How people revisit Web pages: Empirical findings and implications for the design of history systems.” International Journal of Human Computer Studies, 47(1): 97-138, 1997.
Watt, W.C. “Habitability.” American Documentation, July (1968) 338-51.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer
About this chapter
Cite this chapter
Ogden, W., Mcdonald, J., Bernick, P., Chadwick, R. (2008). Habitability In Question-Answering Systems. In: Strzalkowski, T., Harabagiu, S.M. (eds) Advances in Open Domain Question Answering. Text, Speech and Language Technology, vol 32. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-4746-6_15
Download citation
DOI: https://doi.org/10.1007/978-1-4020-4746-6_15
Publisher Name: Springer, Dordrecht
Print ISBN: 978-1-4020-4744-2
Online ISBN: 978-1-4020-4746-6
eBook Packages: Humanities, Social Sciences and LawSocial Sciences (R0)