Advertisement

Towards Automatic Evaluation of Learning Object Metadata Quality

  • Xavier Ochoa
  • Erik Duval
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4231)

Abstract

Thanks to recent developments on automatic generation of metadata and interoperability between repositories, the production, management and consumption of learning object metadata is vastly surpassing the human capacity to review or process these metadata. However, we need to make sure that the presence of some low quality metadata does not compromise the performance of services that rely on that information. Consequently, there is a need for automatic assessment of the quality of metadata, so that tools or users can be alerted about low quality instances. In this paper, we present several quality metrics for learning object metadata. We applied these metrics to a sample of records from a real repository and compared the results with the quality assessment given to the same records by a group of human reviewers. Through correlation and regression analysis, we found that one of the metrics, the text information content, could be used as a predictor of the human evaluation. While this metric is not a definitive measurement of the “real” quality of the metadata record, we present several ways in which it can be used. We also propose new research in other quality dimensions of the learning object metadata.

Keywords

Quality Metrics Human Evaluation Automatic Evaluation Metadata Record Learn Object Metadata 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    IEEE: IEEE Standard for Learning Object Metadata (2002), http://ltsc.ieee.org/doc/wg12/
  2. 2.
    Singh, A., Boley, H., Bhavsar, V.C.: A learning object metadata generator applied to computer science terminology. Presented at the Learning Objects Summit, March 29-30 (2004)Google Scholar
  3. 3.
    Cardinels, K., Meire, M., Duval, E.: Automating metadata generation: the simple indexing interface. In: Proceedings of the 14th WWW conference, pp. 548–556. ACM Press, New York (2005)CrossRefGoogle Scholar
  4. 4.
    Simon, B., Massart, D., van Assche, F., Ternier, S., Duval, E., Brantner, S., et al.: A Simple Query Interface for Interoperable Learning Repositories. In: Simon, B., Olmedilla, D., Saito, N. (eds.), pp. 11–18 (2006)Google Scholar
  5. 5.
    GLOBE. Global Learning Objects Brokered Exchange (2006), http://taste.merlot.org/initiatives/globe.htm
  6. 6.
    Verbert, K., Jovanovic, J., Gaševic, D., Duval, E.: Repurposing Learning Object Components. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM-WS 2005. LNCS, vol. 3762, pp. 1169–1178. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  7. 7.
    Currier, S., Barton, J., O’Beirne, R., Ryan, B.: Quality assurance for digital learning object repositories: issues for the metadata creation process. ALT-J, Research in Learning Technology 12, 5–20 (2004)Google Scholar
  8. 8.
    Barton, J., Currier, S., Hey, J.: Building Quality Assurance into Metadata Creation: an Analysis based on the Learning Objects and e-Prints Communities of Practice. In: Proceedings 2003 Dublin Core Conference, Seatle, Washington, pp. 39–48 (2003)Google Scholar
  9. 9.
    Guy, M., Powell, A., Day, M.: Improving the quality of metadata in Eprint archives, Ariadne (2004)Google Scholar
  10. 10.
    Weibel, S.: Border Crossings: Reflections on a Decade of Metadata Consensus Building. D-Lib Magazine 11 (2005)Google Scholar
  11. 11.
    Ochoa, X., Duval, E.: Quality Metrics for Learning Object Metadata. In: Proceedings ED-Media 2006, pp. 1004–1011 (2006)Google Scholar
  12. 12.
    Bruce, T., Hillman, D.: The continuum of metadata quality: defining, expressing, exploiting. In: Hillman, D., Westbrooks, L. (eds.) Metadata in Practice. American Library Association, Chicago (2004)Google Scholar
  13. 13.
    Duval, E.: LearnRank: the real quality measure for learning materials. In: McCluskey, A. (ed.) Policy and Innovation in Education - Quality Criteria, European Schoolnet, pp. 26–29 (2005)Google Scholar
  14. 14.
    Resnik, P.: Using Information Content to Evaluate Semantic Similarity in a Taxonomy. In: Proceedings of International Joint Conferences on Artificial Intelligence, pp. 448–453 (1995)Google Scholar
  15. 15.
    Aizawa, A.: An information-theoretic perspective of tf–idf measures. Information Processing and Management 39, 45–65 (2003)MATHCrossRefGoogle Scholar
  16. 16.
    Foltz, P., Kintsch, W., Landauer, T.: The measurement of textual coherence with Latent Semantic Analysis. Discourse Processes 25, 285–307 (1998)CrossRefGoogle Scholar
  17. 17.
    Ariadne Foundation. Ariadne Foundation (2005), http://www.ariadne-eu.org
  18. 18.
    Najjar, J., Ternier, S., Duval, E.: User Behavior in Learning Objects Repositories: An Empirical Analysis. In: Proceedings of ED-MEDIA 2004 (2004)Google Scholar
  19. 19.
    Shrout, P., Fleiss, J.: Intraclass Correlations: Uses in Assessing Rater Reliability. Psychological Bulletin 2, 420–428 (1979)CrossRefGoogle Scholar
  20. 20.
    Bederson, B., Shneiderman, B., Wattenberg, M.: Ordered and Quantum Treemaps: Making Effective Use of 2D Space to Display Hierarchies. ACM Transactions on Graphics (TOG) 21, 833–854 (2002)CrossRefGoogle Scholar
  21. 21.
    Shreeves, S., Knutson, E., Stvilia, B., Palmer, C., Twidale, M., Cole, T.: Is “Quality” Metadata "Shareable" Metadata? The Implications of Local Metadata Practices for Federated Collections. In: Proceedings of the Twelfth National Conference of the Association of College and Research Libraries, Association of College and Research Libraries, IL (2005)Google Scholar
  22. 22.
    Lei, Y., Sabou, M., Lopez, V., Zhu, J., Uren, V., Motta, E.: An Infrastructure for Acquiring High Quality Semantic Metadata. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 230–244. Springer, Heidelberg (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Xavier Ochoa
    • 1
  • Erik Duval
    • 2
  1. 1.Escuela Superior Politcnica del Litoral (ESPOL)GuayaquilEcuador
  2. 2.Computerwetenschappen Dept.Katholieke Universiteit LeuvenLeuvenBelgium

Personalised recommendations