An Online-Updating Approach on Task Recommendation in Crowdsourcing Systems

  • Man-Ching YuenEmail author
  • Irwin King
  • Kwong-Sak Leung
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9947)


In crowdsourcing systems, task recommendation can help workers to find their right tasks faster as well as help requesters to receive good quality output quicker. A number of previous works adopted active learning for task recommendation in crowdsourcing systems to achieve certain accuracy with a very low cost. However, the model updating methods in previous works are not suitable for real-world applications. In our paper, we propose a generic online-updating method for learning a factor analysis model, ActivePMF on TaskRec (Probabilistic Matrix Factorization with Active Learning on Task Recommendation Framework), for crowdsourcing systems. The larger the profile of a worker (or task) is, the less important is retraining its profile on each new work done. In case of the worker (or task) having large profile, our algorithm only retrains the whole feature vector of the worker (or task) and keeps all other entries in the matrix fixed. Besides, our algorithm runs batch update to further improve the performance. Experiment results show that our online-updating approach is accurate in approximating to a full retrain while the average runtime of model update for each work done is reduced by more than 90 % (from a few minutes to several seconds).


Crowdsourcing System Recommendation Task Full Retraining Probabilistic Matrix Factorization (PMF) Batch Update 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research was in part supported by grants from the National Grand Fundamental Research 973 Program of China (No. 2014CB340405), the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CUHK 14203314), and Microsoft Research Asia Regional Seed Fund in Big Data Research (Grant No. FY13-RES-SPONSOR-036).


  1. 1.
    Howe, J.: The rise of crowdsourcing. Wired 14(6), 1–4 (2006)Google Scholar
  2. 2.
    Jung, H.J.: Quality assurance in crowdsourcing via matrix factorization based task routing. In: International Conference on World Wide Web (2014)Google Scholar
  3. 3.
    Jung, H.J., Lease, M.: Improving quality of crowdsourced labels via probabilistic matrix factorization. In: Human Computation Workshop at the 26th AAAI (2012)Google Scholar
  4. 4.
    Karimi, R., Freudenthaler, C., Nanopoulos, A., Schmidt-Thieme, L.: Active learning for aspect model in recommender systems. In: Proceedings of IEEE CIDM 2011, pp. 162–167 (2011)Google Scholar
  5. 5.
    Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: Croft, B.W., van Rijsbergen, C.J. (eds.) SIGIR 1994, pp. 3–12. Springer, London (1994)Google Scholar
  6. 6.
    Rendle, S., Schmidt-Thieme, L.: Online-updating regularized kernel matrix factorization models for large-scale recommender systems. In: Proceedings of RecSys 2008, pp. 251–258. ACM, New York (2008)Google Scholar
  7. 7.
    Salakhutdinov, R., Mnih, A.: Probabilistic matrix factorization. In: Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, NIPS 2007, Curran Associates Inc. (2007)Google Scholar
  8. 8.
    Yuen, M.-C., King, I., Leung, K.-S.: A survey of crowdsourcing systems. In: SocialCom 2011, pp. 766–773. IEEE Computer Society (2011)Google Scholar
  9. 9.
    Yuen, M.-C., King, I., Leung, K.-S.: TaskRec: probabilistic matrix factorization in task recommendation in crowdsourcing systems. In: Huang, T., Zeng, Z., Li, C., Leung, C.S. (eds.) ICONIP 2012. LNCS, vol. 7664, pp. 516–525. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-34481-7_63 CrossRefGoogle Scholar
  10. 10.
    Yuen, M.-C., King, I., Leung, K.-S.: Probabilistic matrix factorization with active learning for quality assurance in crowdsourcing systems. In: Proceedings of the IADIS International Conference WWW/Internet 2015, Ireland (2015)Google Scholar
  11. 11.
    Yuen, M.-C., King, I., Leung, K.-S.: TaskRec: a task recommendation framework in crowdsourcing systems. Neural Process. Lett. 41(2), 223–238 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringThe Chinese University of Hong KongSha TinHong Kong

Personalised recommendations