Statistical Quality Control for Human-Based Electronic Services
Crowdsourcing in form of human-based electronic services (people services) provides a powerful way of outsourcing tasks to a large crowd of remote workers over the Internet. Research has shown that multiple redundant results delivered by different workers can be aggregated in order to achieve a reliable result. However, existing implementations of this approach are rather inefficient as they multiply the effort for task execution and are not able to guarantee a certain quality level. As a starting point towards an integrated approach for quality management of people services we have developed a quality management model that combines elements of statistical quality control (SQC) with group decision theory. The contributions of the workers are tracked and weighted individually in order to minimize the quality management effort while guaranteing a well-defined level of overall result quality. A quantitative analysis of the approach based on an optical character recognition (OCR) scenario confirms the efficiency and reach of the approach.
Unable to display preview. Download preview PDF.
- 2.le marquis de Condorcet, M., Caritat, A.N.: Essai sur l’application de l’analyse la probabilit des dcisions rendues la pluralit des voix (1785)Google Scholar
- 3.Dawid, A., Skene, A.: Maximum likelihood estimation of observer Error-Rates using the EM algorithm. Journal of the Royal Statistical Society 28(1), 20–28 (1979)Google Scholar
- 4.Dodge, H., Torrey, M.: Additional continuous sampling inspection plans. Industrial Quality Control (7), 7–12 (1951)Google Scholar
- 7.Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk (2010)Google Scholar
- 8.Juran, J., Godfrey, A.: Juran’s Quality Handbook, 5th edn. McGraw-Hill, New York (2000)Google Scholar
- 9.Kern, R., Bauer, C., Thies, H., Satzger, G.: Validating results of human-based electronic services leveraging multiple reviewers. In: Proceedings of the 16th Americas Conference on Information Systems (AMCIS), Lima, Peru (2010) (forthcoming)Google Scholar
- 10.Kern, R., Zirpins, C., Agarwal, S.: Managing quality of Human-Based eServices. In: Feuerlicht, G., Lamersdorf, W. (eds.) ICSOC 2008.December 1st, 2008, Revised Selected Papers, LNCS, vol. 5472, pp. 304–309. Springer, Heidelberg (2009)Google Scholar
- 13.Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: Tools for iterative tasks on mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 29–30 (2009)Google Scholar
- 19.Ross, J., Irani, L., Silberman, M., Zaldivar, A., Tomlinson, B.: Who are the crowdworkers?: shifting demographics in mechanical turk. In: Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2863–2872 (2010)Google Scholar
- 23.Surowiecki, J.: The Wisdom of Crowds, 1st edn. Doubleday, New York (2004)Google Scholar
- 25.Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., Movellan, J.: Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Advances in Neural Information Processing Systems, vol. 22, pp. 2035–2043 (2009)Google Scholar