Quality Assurance for Human-Based Electronic Services: A Decision Matrix for Choosing the Right Approach

  • Robert Kern
  • Hans Thies
  • Cordula Bauer
  • Gerhard Satzger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6385)

Abstract

Crowdsourcing in the form of human-based electronic services provides a powerful way of outsourcing so called human intelligence tasks (HITs) to a large workforce of people over the Internet. Because of the limited control over that workforce, it is challenging to ensure the quality of the work results. Several approaches have been proposed that can be applied to specific types of HITs. However, it is difficult to identify a suitable quality management approach for any given type of HIT. This paper aims to provide a first sketch of a decision matrix.

References

  1. 1.
    Barr, J., Cabrera, L.F.: AI gets a brain. ACM Queue 4(4), 24–29 (2006)CrossRefGoogle Scholar
  2. 2.
    Kern, R., Bauer, C., Thies, H., Satzger, G.: Validating results of human-based electronic services leveraging multiple reviewers. In: Proceedings of the 16th Americas Conference on Information Systems (AMCIS), Lima, Peru (2010) (forthcoming)Google Scholar
  3. 3.
    Kern, R., Zirpins, C., Agarwal, S.: Managing quality of Human-Based eServices. In: Feuerlicht, G., Lamersdorf, W. (eds.) Service-Oriented Computing - ICSOC 2008 Workshops. LNCS, vol. 5472, pp. 304–309. Springer, Heidelberg (2008)Google Scholar
  4. 4.
    Rowe, G., Wright, G.: The delphi technique as a forecasting tool: issues and analysis. International journal of forecasting 15(4), 353–375 (1999)CrossRefGoogle Scholar
  5. 5.
    Snow, R., OConnor, B., Jurafsky, D., Ng, A.Y.: Cheap and fastbut is it good? evaluating non-expert annotations for natural language tasks. In: EMNLP 2008: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 254–263. ACL, Stroudsburg (2008)CrossRefGoogle Scholar
  6. 6.
    Sorokin, A., Forsyth, D.: Utility data annotation with amazon mechanical turk. In: CVPRW 2008: Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8. IEEE Computer Society, Washington (June 2008)Google Scholar
  7. 7.
    Surowiecki, J.: The Wisdom of Crowds, 1st edn. Doubleday, New York (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Robert Kern
    • 1
  • Hans Thies
    • 1
  • Cordula Bauer
    • 1
  • Gerhard Satzger
    • 1
  1. 1.Karlsruhe Service Research InstituteKarlsruhe Institute of Technology (KIT)KarlsruheGermany

Personalised recommendations