Skip to main content
SpringerLink
Log in
Book cover

Services Computing for Language Resources pp 75–89Cite as

Optimizing Crowdsourcing Workflow for Language Services

  • Chapter
  • First Online:

  • 451 Accesses

Part of the book series: Cognitive Technologies ((COGTECH))

Abstract

Recently, attempts have been to request translation services from anonymous crowds. Compared with platform-based language services, services performed by humans have the advantages of flexibility and quality. However, due to the nature of human services the results are not consistent and quality is not assured. This research tries to solve this problem by creating workflows that make collaboration among crowdsourcing workers far more effective and efficient. We model workers and tasks, and calculate the optimal workflow. To confirm the feasibility of this model, we conduct a computational experiment to calculate the best workflow under various parameters. The results are consistent with existing research so the model is useful in understanding crowdsourcing workflows. In addition, a system is developed for realizing crowdsourcing workflows for translation services. Finally, we develop a translation interface that demonstrates the feasibility of the proposed method.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://langrid.org/tools/toolbox/.

References

  1. Ambati, V., Vogel, S., Carbonell, J.G.: Active learning and crowd-sourcing for machine translation. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10), vol. 11, pp. 2169–2174 (2010)

    Google Scholar 

  2. Aziz, W., Castilho, S., Specia, L.: Pet: a tool for post-editing and assessing machine translation. In: LREC, pp. 3982–3987 (2012)

    Google Scholar 

  3. Aziz, W., Castilho, S., Specia, L.: Pet: a tool for post-editing and assessing machine translation. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pp. 3982–3987 (2012)

    Google Scholar 

  4. Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: Proceedings of the 23rd Annual ACM symposium on User interface software and technology, pp. 313–322. ACM (2010)

    Google Scholar 

  5. Dai, P., Lin, C.H., Weld, D.S.: Pomdp-based control of workflows for crowdsourcing. Artif. Intell. 202, 52–85 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  6. Green, S., Heer, J., Manning, C.D.: The efficacy of human post-editing for language translation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 439–448. ACM (2013)

    Google Scholar 

  7. Ishida, T. (ed.): The Language Grid: Service-Oriented Collective Intelligence for Language Resource Interoperability. Springer Science & Business Media (2011)

    Google Scholar 

  8. Kamar, E., Hacker, S., Horvitz, E.: Combining human and machine intelligence in large-scale crowdsourcing. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems—Volume 1, AAMAS ’12, pp. 467–474. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2012)

    Google Scholar 

  9. Kittur, A., Khamkar, S., André, P., Kraut, R.: Crowdweaver: Visually managing complex crowd work. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12, pp. 1033–1036. ACM, New York, NY, USA (2012)

    Google Scholar 

  10. Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., Horton, J.: The future of crowd work. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13, pp. 1301–1318. ACM, New York, NY, USA (2013)

    Google Scholar 

  11. Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 43–52. ACM (2011)

    Google Scholar 

  12. Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12, pp. 1003–1012. ACM, New York, NY, USA (2012)

    Google Scholar 

  13. Lin, D., Ishida, T.: Participatory service design based on user-centered QoS. In: 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), vol. 1, pp. 465–472 (2013)

    Google Scholar 

  14. Lin, D., Murakami, Y., Ishida, T., Murakami, Y., Tanaka, M.: Composing human and machine translation services: language grid for improving localization processes. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10), vol. 10, pp. 500–506 (2010)

    Google Scholar 

  15. Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Exploring iterative and parallel human computation processes. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10, pp. 68–76. ACM, New York, NY, USA (2010)

    Google Scholar 

  16. Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology, UIST ’10, pp. 57–66. ACM, New York, NY, USA (2010)

    Google Scholar 

  17. Luther, K., Hahn, N., Dow, S.P., Kittur, A.: Crowdlines: supporting synthesis of diverse information sources through crowdsourced outlines. In: Third AAAI Conference on Human Computation and Crowdsourcing, pp. 110–119 (2015)

    Google Scholar 

  18. Morita, D., Ishida, T.: Designing protocols for collaborative translation. In: Proceedings of the 12th International Conference on Principles of Practice in Multi-Agent Systems (PRIMA’09), pp. 17–32. Springer (2009)

    Google Scholar 

  19. Murakami, Y., Lin, D., Ishida, T.: Service-Oriented Architecture for Interoperability of Multilanguage Services, pp. 313–328. Springer, Berlin (2014)

    Google Scholar 

  20. Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 614–622. ACM (2008)

    Google Scholar 

  21. Yoon, K.P., Hwang, C.L.: Multiple Attribute Decision Making: An Introduction, vol. 104. Sage Publications (1995)

    Google Scholar 

  22. Zaidan, O., Callison-Burch, C.: Crowdsourcing translation: professional quality from non-professionals. In: ACL, pp. 1220–1229 (2011)

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by a Grant-in-Aid for Scientific Research (S) (24220002, 2012-2016) from Japan Society for the Promotion of Science (JSPS), Scientific Research (A) (17H00759, 2017-2020) from Japan Society for the Promotion of Science (JSPS), and Service Science, Solutions and Foundation Integrated Research Program from JST RISTEX.

Author information

Authors and Affiliations

  1. Department of Social Informatics, Kyoto University, Kyoto, 606-8501, Japan

    Shinsuke Goto, Toru Ishida & Donghui Lin

Authors
  1. Shinsuke Goto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Toru Ishida
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donghui Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shinsuke Goto .

Editor information

Editors and Affiliations

  1. Kyoto University , Kyoto, Japan

    Yohei Murakami

  2. Kyoto University , Kyoto, Japan

    Donghui Lin

  3. Kyoto University, Kyoto, Japan

    Toru Ishida

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Goto, S., Ishida, T., Lin, D. (2018). Optimizing Crowdsourcing Workflow for Language Services. In: Murakami, Y., Lin, D., Ishida, T. (eds) Services Computing for Language Resources . Cognitive Technologies. Springer, Singapore. https://doi.org/10.1007/978-981-10-7793-7_5

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-7793-7_5

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-7792-0

  • Online ISBN: 978-981-10-7793-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation