Towards Crowdsourcing and Cooperation in Linguistic Resources
Linguistic resources can be populated with data through the use of such approaches as crowdsourcing and gamification when motivated people are involved. However, current crowdsourcing genre taxonomies lack the concept of cooperation, which is the principal element of modern video games and may potentially drive the annotators’ interest. This survey on crowdsourcing taxonomies and cooperation in linguistic resources provides recommendations on using cooperation in existent genres of crowdsourcing and an evidence of the efficiency of cooperation using a popular Russian linguistic resource created through crowdsourcing as an example.
KeywordsGames with a purpose Mechanized labor Wisdom of the crowd Gamification Crowdsourcing Cooperation Linguistic resources
This work is supported by the Russian Foundation for the Humanities, project no. 13-04-12020 “New Open Electronic Thesaurus for Russian”, and by the Program of Government of the Russian Federation 02.A03.21.0006 on 27.08.2013.
The author would like to thank Dmitry Granovsky for the extended statistical information collected from http://opencorpora.org/. The author is also grateful to the anonymous referees who offered very useful comments on the present paper.
- 2.Sabou, M., Bontcheva, K., Derczynski, L., Scharl, A.: Corpus annotation through crowdsourcing: towards best practice guidelines. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), European Language Resources Association (ELRA), pp. 859–866 (2014)Google Scholar
- 4.Quinn, A.J., Bederson, B.B.: A Taxonomy of Distributed Human Computation. Human-Computer Interaction Lab Tech Report, University of Maryland (2009)Google Scholar
- 5.Yuen, M.C., Chen, L.J., King, I.: A survey of human computation systems. In: International Conference on Computational Science and Engineering, 2009. CSE 2009. vol. 4, pp. 723–728. IEEE (2009)Google Scholar
- 6.Sabou, M., Bontcheva, K., Scharl, A.: Crowdsourcing Research Opportunities: Lessons from Natural Language Processing. In: Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies, pp. 17:1–17:8. ACM (2012)Google Scholar
- 9.Erickson, T.: Some thoughts on a framework for crowdsourcing. In: CHI 2011 Workshop on Crowdsourcing and Human Computation (2011)Google Scholar
- 10.Suendermann, D., Pieraccini, R.: Crowdsourcing for Industrial Spoken Dialog Systems. In: Eskénazi, M., Levow, G.A., Meng, H., Parent, G., Suendermann, D. (eds.) Crowdsourcing for Speech Processing: Applications to Data Collection, pp. 280–302. Transcription and Assessment. John Wiley & Sons, Ltd (2013)Google Scholar
- 12.Kohn, A.: No Contest: A Case Against Competition. Houghton Mifflin Harcourt, New York (1992)Google Scholar
- 13.Wilkinson, D.M., Huberman, B.A.: Cooperation and quality in wikipedia. In: Proceedings of the 2007 International Symposium on Wikis, pp. 157–164. ACM (2007)Google Scholar
- 14.Arazy, O., Nov, O.: Determinants of wikipedia quality: the roles of global and local contribution inequality. In: Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work, pp. 233–236. ACM (2010)Google Scholar
- 16.Ranj Bar, A., Maheswaran, M.: Case study: integrity of wikipedia articles. In: Confidentiality and Integrity in Crowdsourcing Systems. SpringerBriefs in Applied Sciences and Technology. Springer International Publishing, pp. 59–66 (2014)Google Scholar
- 19.Braslavski, P., Ustalov, D., Mukhin, M.: A spinning wheel for YARN: user interface for a crowdsourced thesaurus. In: Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, pp. 101–104 (2014)Google Scholar
- 20.Bocharov, V., Alexeeva, S., Granovsky, D., Protopopova, E., Stepanova, M., Surikov, A.: Crowdsourcing morphological annotation. In: Computational Linguistics and Intellectual Technologies: papers from the Annual conference “Dialogue”, RGGU, pp. 109–124 (2013)Google Scholar