Abstract
Human computation systems are often the result of extensive lengthy trial-and-error refinements. What we lack is an approach to systematically engineer solutions based on past successful patterns.
In this paper we present the CrowdLang programming framework for engineering complex computation systems incorporating large crowds of networked humans and machines with a library of known interaction patterns. We evaluate CrowdLang by programming a German-to-English translation program incorporating machine translation and a monolingual crowd. The evaluation shows that CrowdLang is able to simply explore a large design space of possible problem-solving programs with the simple variation of the used abstractions. In an experiment involving 1918 different human actors, we show that the resulting translation program significantly outperforms a pure machine translation in terms of adequacy and fluency whilst translating more than 30 pages per hour and approximates the human-translated gold standard to 75%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Malone, T., Laubacher, R., Johns, T.: General management: The age of hyperspecialization. Harvard Business Review 89(7-8), 56–65 (2011)
Malone, T., Laubacher, R., Dellarocas, C.: The collective intelligence genome. MIT Sloan Management Review 51(3), 21–31 (2010)
Quinn, A., Bederson, B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, pp. 1403–1412. ACM (2011)
Law, E., Ahn, L.: Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5(3), 1–121 (2011)
Bernstein, A., Klein, M., Malone, T.: Programming the global brain. Communications of the ACM 55(5), 1–4 (2012)
Bernstein, A., Klein, M., Malone, T.: The process recombinator: a tool for generating new business process ideas. In: Proceedings of the 20th International Conference on Information Systems, pp. 178–192. Association for Information Systems (1999)
Little, G., Chilton, L., Goldman, M., Miller, R.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 57–66. ACM (2010)
Kittur, A., Smus, B., Khamkar, S., Kraut, R.: Crowdforge: Crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 43–52. ACM (2011)
Ahmad, S., Battle, A., Malkani, Z., Kamvar, S.: The jabberwocky programming environment for structured social computing. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 53–64. ACM (2011)
Bernstein, A.: How can cooperative work tools support dynamic group process? bridging the specificity frontier. In: Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, pp. 279–288. ACM (2000)
Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., Horvitz, E.: Human computation tasks with global constraints. In: CHI (2012)
Minder, P., Bernstein, A.: How to translate a book within an hour - towards general purpose programmable human computers with crowdlang. In: ACM Web Science 2012, New York, NY, USA (2012)
Little, G., Chilton, L., Goldman, M., Miller, R.: Exploring iterative and parallel human computation processes. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 68–76. ACM (2010)
Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008)
Noronha, J., Hysen, E., Zhang, H., Gajos, K.: Platemate: crowdsourcing nutritional analysis from food photographs. In: Proc. of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 1–12. ACM (2011)
Malone, T., Crowston, K.: The interdisciplinary study of coordination. ACM Computing Surveys (CSUR) 26(1), 87–119 (1994)
Chase, R., Aquilano, N., Jacobs, F.: Operations management for competitive advantage. McGraw-Hill/Irwin, New York (2006)
Young, H.: An axiomatization of borda’s rule. Journal of Economic Theory 9(1), 43–52 (1974)
Chen, Y., Liem, B., Zhang, H.: An iterative dual pathway structure for speech-to-text transcription. In: Human Computation: Papers from the AAAI Workshop (WS 2011), San Francisco, CA (August 2011)
Bernstein, M., Little, G., Miller, R., Hartmann, B., Ackerman, M., Karger, D., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 313–322. ACM (2010)
Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation (2002)
Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 65 (2005)
Iman, R., Davenport, J.: Approximations of the critical region of the friedman statistic. Technical report, Sandia Labs, Albuquerque, NM, USA, Texas Tech Univ., Lubbock, USA (1979)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Minder, P., Bernstein, A. (2012). CrowdLang: A Programming Language for the Systematic Exploration of Human Computation Systems. In: Aberer, K., Flache, A., Jager, W., Liu, L., Tang, J., Guéret, C. (eds) Social Informatics. SocInfo 2012. Lecture Notes in Computer Science, vol 7710. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35386-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-35386-4_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-35385-7
Online ISBN: 978-3-642-35386-4
eBook Packages: Computer ScienceComputer Science (R0)