Evaluation of the NSDL and Google for Obtaining Pedagogical Resources

  • Frank McCown
  • Johan Bollen
  • Michael L. Nelson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3652)

Abstract

We describe an experiment that measures the pedagogical usefulness of the results returned by the National Science Digital Library (NSDL) and Google. Eleven public school teachers from the state of Virginia (USA) were used to evaluate a set of 38 search terms and search results based on the Standards of Learning (SOL) for Virginia Public Schools. Evaluations of search results were obtained from the NSDL (572 evaluations) and Google (650 evaluations). In our experiments, teachers ranked the links returned by Google as more relevant to the SOL than the links returned by the NSDL. Furthermore, Google’s ranking of educational material also showed some correlation with expert judgments.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Arms, W., Dushay, N., Fulker, D., Lagoze, C.: A Case Study in Metadata Harvesting: The NSDL. Library HiTech 21(2), 228–237 (2003)CrossRefGoogle Scholar
  2. 2.
    AskNSDL. Response by Melodye Campbell (March 20, 2004), https://ask.nsdl.org/default.aspx?from=srch&id=8337
  3. 3.
    Bar-Ilan, J.: Methods for Measuring Search Engine Performance Over Time. Journal of the American Society for Information Science and Technology 53(4), 308–319 (2002)CrossRefGoogle Scholar
  4. 4.
    Bergmark, D.: Collection Synthesis. In: Proceedings from the second ACM/IEEE-CS joint conference on Digital libraries (JCDL 2002), Portland, Oregon, USA, pp. 253–262 (2002)Google Scholar
  5. 5.
    Bergmark, D., Lagoze, C., Sbityakov, A.: Focused Crawls, Tunneling, and Digital Libraries. In: Agosti, M., Thanos, C. (eds.) ECDL 2002. LNCS, vol. 2458, pp. 91–106. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  6. 6.
    Can, F., Nuray, R., Sevdik, A.: Automatic Performance of Web Search Engines. Information Processing and Management 40(3), 495–514 (2004)CrossRefGoogle Scholar
  7. 7.
    Chowdhury, A., Soboroff, A.: Automatic Evaluation of World Wide Web Search Services. Proceedings of the ACM SIGIR conference 25, 421–422 (2002)Google Scholar
  8. 8.
    Chu, H., Rosenthal, M.: Search Engines for the World Wide Web: A Comparative Study and Evaluation Methodology. Proceedings of the ASIS annual meeting 33, 127–135 (1996)Google Scholar
  9. 9.
    Ding, W., Marchionini, G.: A Comparative Study of Web Search Service Performance. In: Proceedings of the 59th Annual Meeting of the American Society for Information Science, Baltimore, MD, USA, vol. 33, pp. 136–142 (1996)Google Scholar
  10. 10.
    Fulker, D.: Metadata Strategies to Address NSDL Objectives. In: Proceedings of the 5th Russian Conference on Digital Libraries (RCDL 2003), St. Petersburg, Russia (2003)Google Scholar
  11. 11.
    Gordon, M., Pathak, P.: Finding Information on the World Wide Web: The Retrieval Effectiveness of Search Engines. Information Processing and Management 35(2), 141–180 (1999)CrossRefGoogle Scholar
  12. 12.
    Hartson, H., Shivakumar, P., Pérez-Quiñones, M.: Usability Inspection of Digital Libraries: A Case Study. International Journal of Digital Libraries 4(2), 108–123 (2003)CrossRefGoogle Scholar
  13. 13.
    Hawking, D., Craswel, N., Bailey, P., Griffiths, K.: Measuring Search Engine Quality. Information Retrieval 4(1), 33–59 (2001)MATHCrossRefGoogle Scholar
  14. 14.
    Lagoze, C., Hoehn, W., Millman, D., Arms, W., Gan, S., Hillmann, I.C., Krafft, D., Marisa, R., Phipps, J., Saylor, J., Terrizzi, C., Allan, J., Guzman-Lara, S., Kalt, T.: Core Services in the Architecture of the National Science Digital Library (NSDL). In: Proceedings from the second ACM/IEEE-CS joint conference on Digital libraries (JCDL 2002), Portland, Oregon, USA, pp. 201–209 (2002)Google Scholar
  15. 15.
    Leighton, H., Srivastave, J.: First 20 Precision Among World Wide Web Search Services (Search Engines). Journal of American Society for Information Science 50(10), 870–881 (1999)CrossRefGoogle Scholar
  16. 16.
    Ljosland, M.: Evaluation of Web Search Engines and the Search for Better Ranking Algorithms. In: SIGIR 1999 Workshop on Evaluation of Web Retrieval (1999)Google Scholar
  17. 17.
    NSDL Collection Development Policy Draft v.030715. Accessed 1-18-2005, http://content.comm.nsdl.org/doc_tracker/docs_download.php?id=452
  18. 18.
    Silverstein, C., Henzinger, M., Marais, J., Moricz, M.: Analysis of a Very Large Alta Vista Query Log. Technical Report 1998-014. COMPAQ Systems Research Center, Palo Alto, CA (1998)Google Scholar
  19. 19.
    Spink, A., Wolfram, D., Jansen, B., Seracevic, T.: Searching the Web: The Public and Their Queries. Journal of the American Society for Information Science and Technology 52(3), 226–234 (2001)CrossRefGoogle Scholar
  20. 20.
    Standards of Learning for Virginia Public Schools, http://www.pen.k12.va.us/VDOE/Superintendent/Sols/home.shtml
  21. 21.
    Su, L.: A Comprehensive and Systematic Model of User Evaluation of Web Search Engines: I. Theory and Background. Journal of the American Society for Information Science and Technology 54(13), 1175–1192 (2003)CrossRefGoogle Scholar
  22. 22.
    Su., L., Chen, H., Dong, X.: Evaluation of Web-based Search Engines from an End-user’s Perspective: A pilot study. In: Proceedings of the 61st Annual Meeting of the American Society for Information Science, Pittsburgh, PA, USA, pp. 348–361 (1998)Google Scholar
  23. 23.
    Sumner, T., Khoo, M., Recker, M., Marlino, M.: Understanding Educator Perceptions of “Quality” in Digital Libraries. In: Proceedings of the Third ACM+IEEE Joint Conference on Digital Libraries (JCDL 2003), Houston, Texas, USA, pp. 269–279 (2003)Google Scholar
  24. 24.
    Zia, L.: The NSF National Science, Mathematics, Engineering, and Technology Education Digital Library (NSDL) Program: A Progress Report. D-Lib Magazine 6(10) (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Frank McCown
    • 1
  • Johan Bollen
    • 1
  • Michael L. Nelson
    • 1
  1. 1.Computer Science DepartmentOld Dominion UniversityNorfolkUSA

Personalised recommendations