A knowledge-tracing model of learning from a social tagging system

Abstract

We propose a user model to support personalized learning paths through online material. Our approach is a variant of student modeling using the computer tutoring concept of knowledge tracing. Knowledge tracing involves representing the knowledge required to master a domain, and, from traces of online user behavior, diagnosing user knowledge states as a profile over those elements. The user model is induced from documents tagged by an expert in a social tagging system. Tags identified with “expertise” in a domain can be used to identify a corpus of domain documents. That corpus can be fed to an automated process that distills a topic model representation characteristic of the domain. As a learner navigates and reads online material, inferences can be made about the degree to which topics in the target domain have been learned. We validate this knowledge tracing approach against data from a social tagging study. As part of this evaluation, we match the predictions of the knowledge-tracing model to individual participant responses made to individual question items used to test domain knowledge.

This is a preview of subscription content, log in to check access.

References

  1. Abel, F., Baldoni, M., Baroglio, C., Henze, N., Krause, D., Patti, V.: Context-based ranking in folksonomies. Paper presented at the proceedings of the 20th ACM conference on hypertext and hypermedia (HT ’09), pp. 209–218 (2009)

  2. Abel, F., Herder, E., Houben, G.-J., Henze, N., Krause, D.: Cross-system User Modeling and Personalization on the Social Web. User Model. User Adapt. Interact. (2012). doi:10.1007/s11257-012-9131-2

  3. Aleven V., McLaren B.M., Sewall J., Koedinger K.R.: A new paradigm for intelligent tutoring systems: example-tracing tutors. J. Artif. Intell. Educ 19(2), 105–154 (2009)

    Google Scholar 

  4. Allen, I.E., Seaman, J.: Learning on demand: online education in the United States, 2009: Sloan Consortium (2010)

  5. Anderson J.R.: Learning to program in LISP. Cogn. Sci. 8, 87–129 (1984)

    Article  Google Scholar 

  6. Anderson J.R.: The Adaptive Character of Thought. Lawrence Erlbaum Associates, Hillsdale (1990)

    Google Scholar 

  7. Anderson, J.R.: Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale (1993)

  8. Anderson J.R., Bothell D., Byrne M.D., Douglass S., Lebiere C., Qin Y.: An integrated theory of mind. Psychol Rev 11(4), 1036–1060 (2004)

    Article  Google Scholar 

  9. Anderson J.R., Boyle C.F., Corbett A., Lewis M.W.: Cognitive modelling and intelligent tutoring. Artif. Intell. 42, 7–49 (1990)

    Article  Google Scholar 

  10. Balog, K., Azzopardi, L., de Rijke, M.: Formal models for expert finding in enterprise corpora. Paper presented at the 29th annual ACM SIGIR conference on research and development in information retrieval (SIGIR ’06) (2006)

  11. Bernstein, M., Suh, B., Hong, L., Chen, J., Kairam, S., Chi, E.H.: Interactive topic-based browsing of social status streams. Paper presented at the proceedings of the 23rd symposium on user interface software and technology (UIST ’10) (2010)

  12. Blei D.M., Ng A.Y., Jordan M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)

    MATH  Google Scholar 

  13. Bloom B.: The 2 sigma problem: the search for methods of instruction as effective as one-to-one tutoring. Educ. Res. 13(6), 4–16 (1984)

    Google Scholar 

  14. Brin S., Page L.: The anatomy of a large-scale hypertextual web search engine. Comput. Netw. ISDN Syst. 30(1–7), 107–117 (1998)

    Article  Google Scholar 

  15. Brusilovsky P., Peylo C.: Adaptive and intelligent web-based educational systems. Int. J. Artif. Intell. Educ. 13, 156–169 (2003)

    Google Scholar 

  16. Budura, A., Bourges-Waldegg, D., Riordan, J.: Deriving expertise profiles from tags. Paper presented at the international conference on computational science and engineering (CSE ’09) (2009)

  17. Campbell, C.S., Maglio, P.P., Cozzi, A., Dom, B.: Expertise identification using email communications. Paper presented at the proceedings of the 2003 ACM CIKM international conference on information and knowledge management (CIKM ’03) (2003)

  18. Carbonell J.R.: AI in CAI: an artificial-intelligence approach to computer-assisted instruction. IEEE Trans. Man Mach. Syst. 11(4), 190–202 (1970)

    Article  Google Scholar 

  19. Champ, H.: (2009, August 13, 2010). 4,000,000,000. http://blog.flickr.net/en/2009/10/12/4000000000

  20. Conati C., Gertner A., Vanlehn K.: Using Bayesian networks to manage uncertainty in student modeling. User Model. User Adapt. Interact. 12(4), 371–417 (2002)

    MATH  Article  Google Scholar 

  21. Corbett, A.T.: Cognitive computer tutors: solving the two-sigma problem. Paper presented at the user modeling 2001: 8th international conference, Berlin (2001)

  22. Corbett A.T., Anderson J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User Adapt. Interact. 4(4), 253–278 (1995)

    Article  Google Scholar 

  23. Corbett, A. T., Anderson, J. R., O’Brien, A. T.: Student modeling in the ACT Programming Tutor. In: Nichols P. D., Chipman S. F., Brennan R. L. (eds.) Cognitively diagnostic assessment, pp. 19–41. Lawrence Erlbaum Associates, Hillsdale, NJ (1995)

  24. Delicious.: (2008, August 13, 2010). Oh Happy Day. http://blog.delicious.com/blog/2008/07/oh-happy-day.html

  25. Foltz, P.W., Laham, D., Landauer, T.K.: Automated essay scoring: application to educational technology. Paper presented at the world conference on education, multimedia, hypermedia, and telecommunications, Seattle, WA (1999)

  26. Fox, S., Fallows, D.: Internet health resources. Retrieved December, 2003, from http://www.pewinternet.org/reports/pdfs/PIP_Health_Report_July_2003.pdf (2003, August)

  27. Fox, S., Jones, S.: The social life of health information. Pew Internet & American Life Project. Retrieved June 27, 2009, from http://www.pewinternet.org/Reports/2009/8-The-Social-Life-of-Health-Information.aspx (2009, June 11)

  28. Furnas G.W., Landauer T.K., Gomez L.M., Dumais S.T.: The vocabulary problem in human-system communication. Commun. ACM 30, 964–971 (1987)

    Article  Google Scholar 

  29. Gayo-Avello, D., Brenes D.J.: Overcoming spammers in twitter—-a tale of five algorithms. Paper presented at the Congreso Español de Recuperación de Información (CERI 2010) (2010)

  30. Gena, C., Cena, F., Vernero, F., Grillo, P.: The evaluation of a social adaptive web site for cultural events. User Model. User Adapt. Interact. (2012). doi:10.1007/s11257-012-9129-9

  31. Golder S.A., Huberman B.A.: The structure of collaborative tagging systems. J. Inf. Sci. 32, 198–208 (2006)

    Article  Google Scholar 

  32. Griffiths T.L., Steyvers M., Tenenbaum J.B.: Topics in semantic representation. Psychol. Rev. 114(2), 211–244 (2007)

    Article  Google Scholar 

  33. Han S.-G., Lee S.-G., Jo G.-S.: Case-based tutoring systems for procedural problem solving on the www. Expert Syst. Appl. Int. J. 29(3), 573–582 (2005)

    Article  Google Scholar 

  34. Hartley J.R., Sleeman D.H.: Towards more intelligent teaching systems. Int. J. Man Mach. Stud. 5(2), 215–236 (1973)

    Article  Google Scholar 

  35. Hofman, T.: Probabilistic latent semantic indexing. Paper presented at the twenty-second international SIGIR conference on research and development in information retrieval (SIGIR-99) (1999)

  36. Hong, L., Chi, E. H., Budiu, R., Pirolli, P., Nelson, L.: SparTag.us: a low cost tagging system for foraging of web content. Paper presented at the working conference on advanced visual interfaces (AVI ’08) (2008)

  37. Horrigan, J.: The Internet as a resource for news and information about science. Pew Internet & American Life Project. Retrieved June 27, 2009, from http://www.pewinternet.org/Reports/2006/The-Internet-as-a-Resource-for-News-and-Information-about-Science.aspx (2006, November 20)

  38. Hotho, A., Jäschke, R., Schmitz, C., Stumme, G.: Information retrieval in folksonomies: search and ranking. Paper presented at the 3rd European semantic web conference (ESWC ’06) (2006)

  39. John, A., Seligmann, D.: Collaborative tagging and expertise in the enterprise. Paper presented at the collaborative web tagging workshop in conjunction with 15th international conference on world wide web (WWW ’06) (2006)

  40. Junker, B.: Some statistical models and computational methods that may be useful for cognitively-relevant assessment. National Research Council (1999)

  41. Kakkonen, T., Myller, N., Timonen, J., Sutinen, E.: Automatic essay grading with probabilistic latent semantic analysis. Paper presented at the proceedings of the second workshop on building educational applications Using NLP (2005)

  42. Kammerer, Y., Nairn, R., Pirolli, P., Chi, E. H.: Signpost from the masses: learning effects in an exploratory social tag search browser. Paper presented at the 27th conference on human factors in computing systems (CHI ’09) (2009)

  43. Kim, H.-N., El Saddik, A.: Exploring social tagging for personalized community recommendations. User Model. User Adapt. Interact. (2012). doi:10.1007/s11257-012-9130-3

  44. Kleinberg J.: Authoritative sources in a hyperlinked environment. J. ACM 46(5), 604–632 (1999)

    MathSciNet  MATH  Article  Google Scholar 

  45. Kline T.J.B.: Psychological Testing: A Practical Approach to Design and Evaluation. Sage Publications, Thousand Oaks (2005)

    Google Scholar 

  46. Koedinger K.R., Anderson J.R., Hadley W.H., Mark M.A.: Intelligent tutoring goes to the big city. Int. J. Artif. Intell. Educ. 8, 30–43 (1997)

    Google Scholar 

  47. Landauer T.K., Dumais S.T.: A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol. Rev. 104, 211–240 (1997)

    Article  Google Scholar 

  48. Lenhart, A.: Adults and social network websites. Pew Internet & American Life Project. Retrieved June 27, 2009, from http://www.pewinternet.org/Reports/2009/Adults-and-Social-Network-Websites.aspx (2009, January 14)

  49. Loizou, S. K., Dimitrova, V.: Adaptive notifications to support knowledge sharing in close-knit virtual communities. User Model. User Adapt. Interact. (2012). doi:10.1007/s11257-012-9127-y

  50. Mimno, D., McCallum, A.: Expertise modeling for matching papers with reviewers. Paper presented at the 13th international conference on knowledge discovery and data mining (KDD ’07) (2007)

  51. Murray T.: Authoring intelligence tutoring systems: an analysis of the state of the art. Int. J. Artif. Intell. Educ. 10, 98–129 (1999)

    Google Scholar 

  52. Murray T.: An overview of intelligent tutoring system authoring tools: updated analysis of the state of the art. In: Murray, T., Blessing, S., Aisworth, S. (eds) Authoring Tools for Advanced Technology Environments, pp. 493–546. Kluwer, Dordrecht (2003)

    Google Scholar 

  53. Nelson, L., Held, C., Pirolli, P., Hong, L., Schiano, D., Chi, E. H.: With a little help from my friends: examining the impact of social annotations in sensemaking tasks. Paper presented at the proceedings of the 27th international conference on human factors in computing systems (2009)

  54. Noll, M. G., Au Yeung, C.-M., Gibbins, N., Meinel, C., Shadbolt, N.: Telling experts from spammers: expertise ranking in folksonomies. Paper presented at the 32nd annual ACM SIGIR conference on research and development in information retrieval (SIGIR ’09) (2009)

  55. Noll, M. G., Yeung, C.-m. A., Gibbins, N., Meinel, C., Shadbolt, N.: Telling experts from spammers: expertise ranking in folksonomies. Paper presented at the proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval (2009)

  56. PACT.: Pittsburgh advanced cognitive tutor center home. Retrieved August 13, 2010, from http://pact.cs.cmu.edu (2005)

  57. Petkova, D., Croft, B.W.: Hierarchical language models for expert finding in enterprise corpora. Paper presented at the 18th IEEE conference on tools with artificial intelligence (ICTAI ’06) (2006)

  58. Pirolli P., Wilson M.: A theory of the measurement of knowledge content, access, and learning. Psychol. Rev. 105, 58–82 (1998)

    Article  Google Scholar 

  59. Robu V., Halpin H., Shepherd H.: Emergence of consensus and shared vocabularies in collaborative tagging systems. ACM Trans. Web 3(4), 1–34 (2009)

    Article  Google Scholar 

  60. Rosen-Zvi, M., Griffiths, T.L., Steyvers, M., Smyth, P.: The author-topic model for authors and documents. Paper presented at the 20th conference on uncertainty in artificial intelligence (2004)

  61. Shapira, B., Rokach, L., Freilichman, S.: Utilizing facebook single and cross domain data for recommendation systems. User Model. User Adapt. Interact. (2012). doi:10.1007/s11257-012-9128-x

  62. Shute V.J., Psotka J.: Intelligent tutoring systems: past, present, and future. In D.H. Jonassen (ed.) Handbook of Research on Educational Communications and Technology, pp. 570–600. Simon & Schuster Macmillan, New York (1996)

  63. Steyvers M., Griffiths, T.L., Dennis, S.: Probabilistic inference in human semantic memory. TRENDS Cogn. Sci. 10(7), 327–334 (2006)

    Google Scholar 

  64. Weng, J., Lim, E.-P., Jiang, J., He, Q.: TwitterRank: finding topic-sensitive influential twitterers. Paper presented at the proceedings of the 3rd ACM international conference on web search and data mining (WSDM ’10) (2010)

  65. Yin, Z., Li, R., Mei, Q., Han, J.: Exploring social tagging graph for web object classification. Paper presented at the 15th ACM SIGKDD conference on knowledge discovery and data mining (KDD ’09) (2009)

  66. YouTube. YouTube Statistics. Retrieved January 16, 2012, from http://www.youtube.com/t/press_statistics

  67. Zhang, J., Ackerman, M.S.: Searching for expertise in social networks: a simulation of potential strategies. Paper presented at the 2005 international ACM SIGGROUP conference on supporting group work (Group ’05) (2005)

  68. Zhang, J., Ackerman, M. S., Adamic, L.: Expertise networks in online communities: structure and algorithms. Paper presented at the 16th international world wide web conference (WWW ’07) (2007)

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Peter Pirolli.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Pirolli, P., Kairam, S. A knowledge-tracing model of learning from a social tagging system. User Model User-Adap Inter 23, 139–168 (2013). https://doi.org/10.1007/s11257-012-9132-1

Download citation

Keywords

  • Cognitive models
  • User models
  • Latent Dirichlet allocation
  • LDA
  • Topic models
  • SparTag.us
  • Social tagging
  • Social web