CrowdLang: A Programming Language for the Systematic Exploration of Human Computation Systems

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7710)


Human computation systems are often the result of extensive lengthy trial-and-error refinements. What we lack is an approach to systematically engineer solutions based on past successful patterns.

In this paper we present the CrowdLang programming framework for engineering complex computation systems incorporating large crowds of networked humans and machines with a library of known interaction patterns. We evaluate CrowdLang by programming a German-to-English translation program incorporating machine translation and a monolingual crowd. The evaluation shows that CrowdLang is able to simply explore a large design space of possible problem-solving programs with the simple variation of the used abstractions. In an experiment involving 1918 different human actors, we show that the resulting translation program significantly outperforms a pure machine translation in terms of adequacy and fluency whilst translating more than 30 pages per hour and approximates the human-translated gold standard to 75%.


CrowdLang Programming Language Human Computation Collective Intelligence Crowdsourcing Translation Software 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Malone, T., Laubacher, R., Johns, T.: General management: The age of hyperspecialization. Harvard Business Review 89(7-8), 56–65 (2011)Google Scholar
  2. 2.
    Malone, T., Laubacher, R., Dellarocas, C.: The collective intelligence genome. MIT Sloan Management Review 51(3), 21–31 (2010)Google Scholar
  3. 3.
    Quinn, A., Bederson, B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, pp. 1403–1412. ACM (2011)Google Scholar
  4. 4.
    Law, E., Ahn, L.: Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5(3), 1–121 (2011)CrossRefGoogle Scholar
  5. 5.
    Bernstein, A., Klein, M., Malone, T.: Programming the global brain. Communications of the ACM 55(5), 1–4 (2012)CrossRefGoogle Scholar
  6. 6.
    Bernstein, A., Klein, M., Malone, T.: The process recombinator: a tool for generating new business process ideas. In: Proceedings of the 20th International Conference on Information Systems, pp. 178–192. Association for Information Systems (1999)Google Scholar
  7. 7.
    Little, G., Chilton, L., Goldman, M., Miller, R.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 57–66. ACM (2010)Google Scholar
  8. 8.
    Kittur, A., Smus, B., Khamkar, S., Kraut, R.: Crowdforge: Crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 43–52. ACM (2011)Google Scholar
  9. 9.
    Ahmad, S., Battle, A., Malkani, Z., Kamvar, S.: The jabberwocky programming environment for structured social computing. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 53–64. ACM (2011)Google Scholar
  10. 10.
    Bernstein, A.: How can cooperative work tools support dynamic group process? bridging the specificity frontier. In: Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, pp. 279–288. ACM (2000)Google Scholar
  11. 11.
    Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., Horvitz, E.: Human computation tasks with global constraints. In: CHI (2012)Google Scholar
  12. 12.
    Minder, P., Bernstein, A.: How to translate a book within an hour - towards general purpose programmable human computers with crowdlang. In: ACM Web Science 2012, New York, NY, USA (2012)Google Scholar
  13. 13.
    Little, G., Chilton, L., Goldman, M., Miller, R.: Exploring iterative and parallel human computation processes. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 68–76. ACM (2010)Google Scholar
  14. 14.
    Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  15. 15.
    Noronha, J., Hysen, E., Zhang, H., Gajos, K.: Platemate: crowdsourcing nutritional analysis from food photographs. In: Proc. of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 1–12. ACM (2011)Google Scholar
  16. 16.
    Malone, T., Crowston, K.: The interdisciplinary study of coordination. ACM Computing Surveys (CSUR) 26(1), 87–119 (1994)CrossRefGoogle Scholar
  17. 17.
    Chase, R., Aquilano, N., Jacobs, F.: Operations management for competitive advantage. McGraw-Hill/Irwin, New York (2006)Google Scholar
  18. 18.
    Young, H.: An axiomatization of borda’s rule. Journal of Economic Theory 9(1), 43–52 (1974)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Chen, Y., Liem, B., Zhang, H.: An iterative dual pathway structure for speech-to-text transcription. In: Human Computation: Papers from the AAAI Workshop (WS 2011), San Francisco, CA (August 2011)Google Scholar
  20. 20.
    Bernstein, M., Little, G., Miller, R., Hartmann, B., Ackerman, M., Karger, D., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 313–322. ACM (2010)Google Scholar
  21. 21.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation (2002)Google Scholar
  22. 22.
    Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 65 (2005)Google Scholar
  23. 23.
    Iman, R., Davenport, J.: Approximations of the critical region of the friedman statistic. Technical report, Sandia Labs, Albuquerque, NM, USA, Texas Tech Univ., Lubbock, USA (1979)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Dynamic and Distributed Information Systems GroupUniversity of ZurichSwitzerland

Personalised recommendations