Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

The role of evaluative metadata in an online teacher resource exchange


A large-scale online teacher resource exchange is studied to examine the ways in which metadata influence teachers’ selection of resources. A hierarchical linear modeling approach was used to tease apart the simultaneous effects of resource features and author features. From a decision heuristics theoretical perspective, teachers appear to rely on complex heuristics that integrate many dimensions when determining whether to download a resource. Most surprisingly, numbers of ratings more strongly predict downloads than do mean rating levels, such that multiple negative ratings appear to attract more downloads than do few positive ratings. Implications for system design are discussed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3


  1. 1.


  2. 2.


  3. 3.


  4. 4.


  5. 5.

    These estimates were calculated using the following procedure: we factor in all of a resource’s attributes using multiple regression coefficients by first adding and subtracting the log-odds of various predictors before exponentiating to determine the predicted number of downloads.


  1. About OER Commons (2013). Retrieved October 9, 2013 from http://www.oercommons.org/about.

  2. Abramovich, S., & Schunn, C. (2012). Studying teacher selection of resources in an ultra-large scale interactive system: Does metadata guide the way? Computers and Education, 58(1), 551–559.

  3. Ball, D., & Cohen, D. (1996). Reform by the book: What is: Or might be: The role of curriculum materials in teacher learning and instructional reform? Educational Researcher, 25(9), 6–14.

  4. Brown, M., & Edelson, D. C. (2003). Teaching as design: Can we better understand the ways in which teachers use materials so we can better design materials to support their changes in practice LeTUS report series. Evanston, IL: The Center for Learning Technologies in Urban Schools.

  5. Caswell, T., Henson, S., Jensen, M., & Wiley, D. (2008). Open content and open educational resources: Enabling universal education. The International Review of Research in Open and Distance Learning, 9(1).

  6. Chiao-Fang, H., Khabiri, E., & Caverlee, J. (2009). Ranking comments on the social web. Paper presented at the International Conference on Computational Science and Engineering.

  7. Clements, K. I., & Pawlowski, J. M. (2012). User-oriented quality for OER: understanding teachers’ views on re-use, quality, and trust. Journal of Computer Assisted Learning, 28(1), 4–14.

  8. Colton, A. B., & Sparks-Langer, G. M. (1993). A conceptual framework to guide the development of teacher reflection and decision making. Journal of Teacher Education, 44(1), 45–54.

  9. Czerlinski, J., Gigerenzer, G., & Goldstein, D. G. (1999). How good are simple heuristics? In G. Gigerenzer, P. M. Todd, & A. R. Group (Eds.), Simple heuristics that make us smart (pp. 97–118). New York, NY: Oxford University Press.

  10. Davis, E. A., & Krajcik, J. S. (2005). Designing educative curriculum materials to promote teacher learning. Educational Researcher, 34(3), 3–14.

  11. Dholakia, U. M., King, W. J., & Baraniuk, R. (2006). What makes an open education program sustainable? The case of connexions. Retrieved October 9, 2013 from https://oerknowledgecloud.org/sites/oerknowledgecloud.org/files/36781781.pdf.

  12. Diekema, A. R., & Olsen, M. W. (2011). Personal information management practices of teachers. Proceedings of the American Society for Information Science and Technology, 48(1), 1–10.

  13. Doyle, W., & Ponder, G. A. (1977). The practicality ethic in teacher decision-making. Interchange, 8(3), 1–12.

  14. El-Hani, C. N., & Greca, I. M. (2013). ComPratica: A virtual community of practice for promoting biology teachers’ Professional Development in Brazil. Research in Science Education, 43(4), 1327–1359.

  15. Fitzgerald, M. A., Lovin, V., & Branch, R. M. (2003). The gateway to educational materials: An evaluation of an online resource for teachers and an exploration of user behavior. Journal of Technology and Teacher Education, 11(1), 21–51.

  16. Gewertz, C. (2012). Educators in Search of Common-Core Resources. Retrieved May 21, 2012 from http://www.edweek.org/ew/articles/2012/02/24/22resources_ep.h31.html.

  17. Gray, L., Thomas, N., & Lewis, L. (2010). Teachers Use of Educational Technology in U.S. Public Schools (2009): National Center for Education Statistics, Institute of Education Sciences. Washington, DC: Department of Education.

  18. Gueudet, G., & Trouche, L. (2012). Teachers’ work with resources: Documentational geneses and professional geneses. In G. Gueudet, B. Pepin, & L. Trouche (Eds.), From text to ‘lived’ resources (pp. 23–41). Dordrecht: Springer.

  19. Hanson, K., & Carlson, B. (2005). Effective access: Teachers’ use of digital resources in STEM teaching. Newton, MA: Education Development Center Inc.

  20. Harris, J., & Hofer, M. (2009). Instructional planning activity types as vehicles for curriculum-based TPACK development. Paper presented at the Society for Information Technology and Teacher Education International Conference 2009, Charleston, SC, USA.

  21. Hennessy, S., Ruthven, K., & Brindley, S. (2005). Teacher perspectives on integrating ICT into subject teaching: commitment, constraints, caution, and change. Journal of Curriculum Studies, 37(2), 155–192.

  22. Hill, J. R., & Hannafin, M. J. (2001). Teaching and learning in digital environments: The resurgence of resource-based learning. Educational Technology Research and Development, 49(3), 37–52.

  23. Klerkx, J., Vandeputte, B., Parra, G., Santos, J., Assche, F., & Duval, E. (2010). How to share and reuse learning resources: The ARIADNE experience. In M. Wolpers, P. Kirschner, M. Scheffel, S. Lindstaedt, & V. Dimitrova (Eds.), Sustaining TEL: From innovation to learning and practice (Vol. 6383, pp. 183–196). Berlin: Springer.

  24. Lau, S.-H., & Woods, P. C. (2009). Understanding learner acceptance of learning objects: The roles of learning object characteristics and individual differences. British Journal of Educational Technology, 40(6), 1059–1075.

  25. Lawless, K. A., & Pellegrino, J. W. (2007). Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research, 77(4), 575–614.

  26. Li, J. Z. (2010). Quality, evaluation and recommendation for learning object. Paper presented at the International Conference on Educational and Information Technology, Chongqing, China.

  27. Maloch, B., Flint, A. S., Eldridge, D., Harmon, J., Loven, R., Fine, J. C., et al. (2003). Understandings, beliefs, and reported decision making of first-year teachers from different reading teacher preparation programs. The Elementary School Journal, 103(5), 431–457.

  28. Manouselis, N., Vuorikari, R., & Van Assche, F. (2010). Collaborative recommendation of e-learning resources: an experimental investigation. Journal of Computer Assisted Learning, 26(4), 227–242.

  29. McCutcheon, G. (1980). How do elementary school teachers plan? The nature of planning and influences on it. The Elementary School Journal, 81(1), 4–23.

  30. Morrison, J. (2010). A research framework to examine the impacts of online peer review on learning outcomes in computer science courses. Paper presented at the ITS 2010 workshop: Computer-supported peer review in education: Synergies with Intelligent Tutoring Systems Workshop, Pittsburgh, PA.

  31. Mudambi, S. M., & Schuff, D. (2010). What makes a helpful online review? A study of customer reviews on Amazon.com. MIS Quarterly, 34(1), 185–200.

  32. Munby, H. (1982). The place of teachers’ beliefs in research on teacher thinking and decision making, and an alternative methodology. Instructional Science, 11(3), 201–225.

  33. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. doi:10.1037/0033-295X.84.3.231.

  34. Ochoa, X., & Duval, E. (2008). Relevance ranking metrics for learning objects. IEEE Transactions on Learning Technologies, 1(1), 34–48.

  35. Ochoa, X., Klerkx, J., Vandeputte, B., & Duval, E. (2011). On the use of learning object metadata: The GLOBE experience. In C. Kloos, D. Gillet, R. Crespo García, F. Wild, & M. Wolpers (Eds.), Towards ubiquitous learning (Vol. 6964, pp. 271–284). Berlin: Springer.

  36. Ozogul, G., Olina, Z., & Sullivan, H. (2008). Teacher, self and peer evaluation of lesson plans written by preservice teachers. Educational Technology Research and Development, 56(2), 181–201.

  37. Perrault, A. M. (2007). An exploratory study of biology teachers’ online information seeking practices. School Library Media Research, 10, 1–26.

  38. Poston, R. S., & Speier, C. (2005). Effective use of knowledge management systems: A process model of content ratings and credibility indicators. MIS Quarterly, 29, 221–244.

  39. Raab, M., & Gigerenzer, G. (2005). Intelligence as smart heuristics. In R. J. Sternberg & J. E. Pretz (Eds.), Cognition and intelligence (pp. 188–207). Cambridge: Cambridge University Press.

  40. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models. Thousand Oaks, CA: Sage.

  41. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., Congdon, R. T., & du Toit, M. (2011). HLM 7. Lincolnwood, IL: Scientific Software International Inc.

  42. Recker, M., Dorward, J., & Nelson, L. (2004). Discovery and use of online learning resources: Case study findings. Educational Technology and Society, 7(2), 93–104.

  43. Recker, M., Sellers, L., & Ye, L. (2012). Investigating impacts of technology-related teacher professional development designs: A comparative case study. Paper 5: The instructional architect research group. Retrieved October 7, 2013 from http://digitalcommons.usu.edu/iagroup/5/.

  44. Recker, M., Walker, A., Giersch, S., Mao, X., Halioris, S., Palmer, B., et al. (2007). A study of teachers’ use of online learning resources to design classroom activities. New Review of Hypermedia and Multimedia, 13(2), 117–134.

  45. Remillard, J. T. (2012). Modes of engagement: Understanding teachers’ transactions with mathematics curriculum resources. In G. Gueudet, B. Pepin, & L. Trouche (Eds.), From text to ‘lived’ resources (pp. 105–122). Dordrecht: Springer.

  46. Schonfeld, E. (2011). Tablet Education Startup ShowMe Raises $800,000 From Lerer Ventures, SV Angel, And Betaworks. Retrieved September 22, 2012 from http://techcrunch.com/2011/08/16/tablet-education-showme-800000/.

  47. Speer, N. M. (2005). Issues of methods and theory in the study of mathematics teachers’ professed and attributed beliefs. Educational Studies in Mathematics, 58(3), 361–391.

  48. Stein, M. K., Grover, B. W., & Henningsen, M. (1996). Building student capacity for mathematical thinking and reasoning: An analysis of mathematical tasks used in reform classrooms. American Educational Research Journal, 33(2), 455–488.

  49. Stern, P., & Shavelson, R. J. (1983). Reading teachers’ judgments, plans, and decision making. The Reading Teacher, 37(3), 280–286.

  50. Sumner, T., Khoo, M., Recker, M., & Marlino, M. (2003). Understanding educator perceptions of quality in digital libraries. Paper presented at the Digital Libraries, 2003. Proceedings. 2003 Joint Conference on.

  51. Tillema, H. H. (2009). Assessment for learning to teach appraisal of practice teaching lessons by mentors, supervisors, and student teachers. Journal of Teacher Education, 60(2), 155–167.

  52. Todd, P. M., & Gigerenzer, G. (2000). Précis of simple heuristics that make us smart. Behavioral and Brain Sciences, 23(5), 727–741.

  53. Tubin, D., & Edri, S. (2004). Teachers planning and implementing ICT-based practices. Planning and Changing, 35(3), 181–191.

  54. Tucker, B. (2011). Teachers swap recipes. Education Next, 11(3), 10–17.

  55. Vargo, J., Nesbit, J. C., Belfer, K., & Archambault, A. (2003). Learning object evaluation: Computer-mediated collaboration and inter-rater reliability. International Journal of Computers and Applications, 25(3), 198–205.

  56. Weiss, C. H., Cambone, J., & Wyeth, A. (1992). Trouble in paradise: Teacher conflicts in shared decision making. Educational Administration Quarterly, 28(3), 350–367.

Download references

Author information

Correspondence to Samuel Abramovich.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Abramovich, S., Schunn, C.D. & Correnti, R.J. The role of evaluative metadata in an online teacher resource exchange. Education Tech Research Dev 61, 863–883 (2013). https://doi.org/10.1007/s11423-013-9317-2

Download citation


  • Online
  • Teaching resources
  • Metadata
  • Ratings
  • Hierarchical linear model
  • Decision heuristics