What you think and what I think: Studying intersubjectivity in knowledge artifacts evaluation
Miscalibration, the failure to accurately evaluate one’s own work relative to others' evaluation, is a common concern in social systems of knowledge creation where participants act as both creators and evaluators. Theories of social norming hold that individual’s self-evaluation miscalibration diminishes over multiple iterations of creator-evaluator interactions and shared understanding emerges. This paper explores intersubjectivity and the longitudinal dynamics of miscalibration between creators' and evaluators' assessments in IT-enabled social knowledge creation and refinement systems. Using Latent Growth Modeling, we investigated dynamics of creator’s assessments of their own knowledge artifacts compared to peer evaluators' to determine whether miscalibration attenuates over multiple interactions. Contrary to theory, we found that creator’s self-assessment miscalibration does not attenuate over repeated interactions. Moreover, depending on the degree of difference, we found self-assessment miscalibration to amplify over time with knowledge artifact creators' diverging farther from their peers' collective opinion. Deeper analysis found no significant evidence of the influence of bias and controversy on miscalibration. Therefore, relying on social norming to correct miscalibration in knowledge creation environments (e.g., social media interactions) may not function as expected.
KeywordsIntersubjectivity Miscalibration Longitudinal analysis Knowledge artifacts Peer-evaluation Latent classes
- Bandura, A. (1962). Social learning through imitation. University of Nebraska Press.Google Scholar
- Bandura, A., & Walters, R. H. (1963). Social Learning and Personality Development. New York, Holt, Rinehart and Winston .Google Scholar
- Bouzidi, L., & Jaillet, A. (2009). Can online peer assessment be trusted? Educational Technology & Society, 12(4), 257–268.Google Scholar
- Campbell, D. J. (1988). Task complexity: a review and analysis. Academy of Management Review, 13(1), 40–52.Google Scholar
- Conklin, J. (2001). Wicked problems and social complexity. Dialogue Mapping:Building Shared Understanding of Wicked Problems Retrieved from http://www.ideapartnership.org/documents/wickedproblems.pdf.Google Scholar
- Crooks, T. (2001). The validity of formative assessments. University of Leeds.Google Scholar
- Dai, H., Zhu, F., Lim, E.-P., & Pang, H. (2012). Detecting Anomalies in Bipartite Graphs with Mutual Dependency Principles. In IEEE 12th International Conference on Data Mining (ICDM) 2012 (pp. 171–180). IEEE. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6413905
- Dede, C. (2008). A seismic shift in epistemology. Educause Review, 43(3).Google Scholar
- Dorst, K. (2003). The problem of design problems. Expertise in Design, 135–147.Google Scholar
- Duncan, T. E. (1999). An introduction to latent variable growth curve modeling: Concepts, issues, and applications. Mahwah, N.J.:L. Erlbaum Associates Retrieved from http://search.ebscohost.com/login.aspx?direct=true&scope=site&db=nlebk&db=nlabk&AN=19332.Google Scholar
- Edwards, K. (2001). Epistemic communities. Situated Learning and Open Source Software Development. Epistemic Cultures and the Practice of Interdisciplinarity Retrieved from http://orbit.dtu.dk/fedora/objects/orbit:51813/datastreams/file_2976336/content.
- Ford, E., & Babik, D. (2013). Methods and Systems for Educational On-Line Methods.Google Scholar
- Gagne, R. M. (1985). The conditions of learning and theory of instruction. Holt Rinehart & Winston.Google Scholar
- Habermas, J. (1981). New social movements.Google Scholar
- Hardaway, D. E., & Scamell, R. W. (2012). Open knowledge creation: bringing transparency and inclusiveness to the peer review process. MIS Quarterly, 36(2).Google Scholar
- Hermida, A. (2011). Social media is inherently a system of peer evaluation [blog]. Retrieved September 21, 2012, from http://blogs.lse.ac.uk/impactofsocialsciences/2011/06/27/social-media-is-inherently-a-system-of-peer-evaluation-and-is-changing-the-way-scholars-disseminate-their-research-raising-questions-about-the-way-we-evaluate-academic-authority/.
- Huhta, A. (2008). Diagnostic and formative assessment. In B. Spolsky, & F. M. Hult (Eds.), The handbook of educational linguistics (pp. 469–482). Blackwell Publishing Ltd. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/9780470694138.ch33/summary.
- Joordens, S., Desa, S., & Paré, D. (2009). The pedagogical anatomy of peer assessment: dissecting a peerScholar assignment. Journal of Systemics, Cybernetics & Informatics, 7(5) Retrieved from http://www.iiisci.org/journal/CV$/sci/pdfs/XE123VF.pdf.
- King, A. (1989). Verbal interaction and problem-solving within computer-assisted cooperative learning groups. Journal of Educational Computing Research, 5(1), 15.Google Scholar
- Kreps, D. M. (1997). Intrinsic motivation and extrinsic incentives. American Economic Review, 87(2), 359–364.Google Scholar
- Krosnick, J. A. (1999). Maximizing questionnaire quality. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of political attitudes (pp. 37–57). San Diego, CA US: Academic Press.Google Scholar
- Krosnick, J. A., Thomas, R., & Shaeffer, E. (2003). How Does Ranking Rate?: A Comparison of Ranking and Rating Tasks. In Conference Papers – American Association for Public Opinion Research (p. N.PAG).Google Scholar
- Lauw, H. W., Lim, E.-P., & Wang, K. (2006). Bias and Controversy: Beyond the Statistical Deviation. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 625–630). ACM. Retrieved from http://dl.acm.org/citation.cfm?id=1150478
- Meyer, B. D. (1995). Natural and quasi-experiments in economics. Journal of Business & Economic Statistics, 13(2).Google Scholar
- Piaget, J., & Gabain, M. (1926). The language and thought of the child, by Jean Piaget...Preface by Professor E. Claparède. London, K. Paul, Trench, Trubner & co., ltd.; New York, Harcourt Brace & company, inc., 1926.Google Scholar
- Raman, K., & Joachims, T. (2014). Methods for Ordinal Peer Grading. arXiv:1404.3656 [cs]. Retrieved from http://arxiv.org/abs/1404.3656
- Roos, M., Rothe, J., Rudolph, J., Scheuermann, B., & Stoyan, D. (2012). A Statistical Approach to Calibrating the Scores of Biased Reviewers: The Linear vs. the Nonlinear Model. In Proceedings of the 6th Multidisciplinary Workshop on Advances in Preference Handling. Retrieved from http://mpref2012.lip6.fr/proceedings/RoosMPREF2012.pdf
- Sadler, P. M., & Good, E. (2006). The impact of self-and peer-grading on student learning. Educational Assessment, 11(1), 1–31.Google Scholar
- Scheff, T. J. (2006). Goffman unbound!: A New paradigm for social science. Boulder, Colo.:Paradigm Publishers.Google Scholar
- Schutz, A. (1967). The Phenomenology of the Social World. Translated by George Walsh and Frederick Lehnert. With an introd. by George Walsh. Evanston, Ill.:Northwestern University Press.Google Scholar
- Shah, N. B., Bradley, J. K., Parekh, A., Wainwright, M., & Ramchandran, K. (2013). A case for ordinal peer evaluation in MOOCs. NIPS Workshop on Data Driven Education Retrieved from http://lytics.stanford.edu/datadriveneducation/papers/shahetal.pdf.
- Shepard, L. A. (2007). Formative assessment: Caveat emptor. Erlbaum.Google Scholar
- Simon, H. A. (1959). Theories of decision-making in economics and behavioral science. The American Economic Review, 49(3), 253–283.Google Scholar
- Simon, H. A. (1969). The sciences of the artificial. The MIT Press.Google Scholar
- Slavin, R. E. (1992). When and why does cooperative learning increase academic achievement? Theoretical and empirical perspectives:Cambridge University Press.Google Scholar
- Sluijsmans, D., Brand-Gruwel, S., van Merriënboer, J. J. G., & Bastiaens, T. J. (2002). The training of peer assessment skills to promote the development of reflection skills in teacher education. Studies in Educational Evaluation, 29(1), 23–42. doi:10.1016/S0191-491X(03)90003-4.CrossRefGoogle Scholar
- Surowiecki, J. (2004). The wisdom of crowds: Why the many are smarter than the Few and How collective wisdom shapes business, economies, societies, and nations. New York:Doubleday.Google Scholar