Converging Human Knowledge for Opinion Mining

  • Jiacheng Liu
  • Feilong TangEmail author
  • Long Chen
  • Liang Qiao
  • Yanqin Yang
  • Wenchao Xu
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 612)


Opinion mining focuses on analyzing opinions in documents. Existing most algorithms for mining opinion either are machine-only, leaving plenty of confused puzzles due to lacking human background knowledge, or using opinion dictionary from domain experts. The latter is expensive and hard to scale. In this paper, we propose a novel approach RULING (conveRging hUman knowLedge opInion miNinG) for opinion mining, where human include both the crowd and the experts. Firstly, we propose a method for combining expert knowledge with the machine learning method. Then we use the prediction result to find out the hard item, and classify them using crowdsourcing. This method can scale better than the previous methods and get a better result. Experimental results demonstrate our RULING approach outperforms related proposals in terms of classification performance.



This work was supported in part by the National Natural Science Foundation of China projects under Grants 91438121, 61373156, 61672351, 61532013 and U1636210, in part by the National Basic Research Program under Grant 2015CB352403, in part by the Huawei Technologies Co. Ltd., project under Grant YBN2016090103, and in part by the National Key Research and Development Program of China under Grant 2016YFB0700502 and the Scientific Innovation Act of STCSM under Grant 15JC1402400.


  1. 1.
    Abbasi, A., Chen, H., Salem, A.: Sentiment analysis in multiple languages: feature selection for opinion classification in web forums. ACM Trans. Inf. Syst. (TOIS) 26(3), 12 (2008)CrossRefGoogle Scholar
  2. 2.
    Agarwal, A., Xie, B., Vovsha, I., Rambow, O., Passonneau, R.: Sentiment analysis of twitter data. In: Proceedings of the Workshop on Languages in Social Media, Association for Computational Linguistics, pp. 30–38 (2011)Google Scholar
  3. 3.
    Aggarwal, C.C., Zhai, C.: Mining Text Data. Springer, New York (2012)CrossRefGoogle Scholar
  4. 4.
    Bishop, C.M.: Pattern recognition. Mach. Learn. 128, 32–38 (2006)Google Scholar
  5. 5.
    Boia, M., Musat, C.C., Faltings, B.: Acquiring commonsense knowledge for sentiment analysis using human computation. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (2014). doi: 10.1145/2567948.2577316
  6. 6.
    Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s mechanical turk a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6(1), 3–5 (2011)CrossRefGoogle Scholar
  7. 7.
    Feyisetan, O., Simperl, E., Van Kleek, M., Shadbolt, N.: Improving paid microtasks through gamification and adaptive furtherance incentives. In: Proceedings of the 24th International Conference on World Wide Web. ACM (2015)Google Scholar
  8. 8.
    Haas, D., Ansel, J., Gu, L., Marcus, A.: Argonaut: macrotask crowdsourcing for complex data processing. Proc. VLDB Endowment 8(12), 1642–1653 (2015)CrossRefGoogle Scholar
  9. 9.
    Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM (2004)Google Scholar
  10. 10.
    Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint (2014). arXiv:1408.5882
  11. 11.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  12. 12.
    Law, E., Yin, M., Goh, J., Chen, K., Terry, M.A., Gajos, K.Z.: Curiosity killed the cat, but makes crowdwork better. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM (2016)Google Scholar
  13. 13.
    Liao, S.H.: Expert system methodologies and applicationsa decade review from 1995 to 2004. Expert Syst. Appl. 28(1), 93–103 (2005)CrossRefGoogle Scholar
  14. 14.
    Liu, B.: Sentiment analysis and opinion mining. Synth. Lect. Hum. Lang. Technol. 5(1), 1–167 (2012)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: CDAS: a crowdsourcing data analytics system. Proc. VLDB Endowment 5(10), 1040–1051 (2012)CrossRefGoogle Scholar
  16. 16.
    Madden, M., Lenhart, A., Cortesi, S., Gasser, U.: Pew Internet and American Life Project. Pew Research Center, Washington, DC (2010)Google Scholar
  17. 17.
    Martineau, J., Finin, T.: Delta TFIDF: an improved feature space for sentiment analysis. ICWSM 9, 106 (2009)Google Scholar
  18. 18.
    Mason, W., Watts, D.J.: Financial incentives and the performance of crowds. ACM SigKDD Explor. Newsl. 11(2), 100–108 (2010)CrossRefGoogle Scholar
  19. 19.
    McAuley, J., Pandey, R., Leskovec, J.: Inferring networks of substitutable and complementary products. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM (2015)Google Scholar
  20. 20.
    Mozafari, B., Sarkar, P., Franklin, M., Jordan, M., Madden, S.: Scaling up crowd-sourcing to very large datasets: a case for active learning. Proc. VLDB Endowment 8(2), 125–136 (2014)CrossRefGoogle Scholar
  21. 21.
    Pennebaker, J.W., Francis, M.E., Booth, R.J.: Linguistic inquiry and word count. In: LIWC 2001, vol. 71. Lawrence Erlbaum Associates, Mahway (2001)Google Scholar
  22. 22.
    Platt, J., et al.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif. 10(3), 61–74 (1999)Google Scholar
  23. 23.
    Shaw, A.D., Horton, J.J., Chen, D.L.: Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work. ACM (2011)Google Scholar
  24. 24.
    Socher, R., Perelygin, A., Wu, J.Y., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: EMNLP, vol. 1631, p. 1642. Citeseer (2013)Google Scholar
  25. 25.
    Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M.: Lexicon-based methods for sentiment analysis. Comput. Linguist. 37(2), 267–307 (2011)CrossRefGoogle Scholar
  26. 26.
    Wilson, S., Schaub, F., Ramanath, R., Sadeh, N., Liu, F., Smith, N.A., Liu, F.: Crowdsourcing annotations for websites’ privacy policies: can it really work? In: Proceedings of the 25th International Conference on World Wide Web, International World Wide Web Conferences Steering Committee (2016)Google Scholar
  27. 27.
    Wu, H., Sun, H., Fang, Y., Hu, K., Xie, Y., Song, Y., Liu, X.: Combining machine learning and crowdsourcing for better understanding commodity reviews. In: AAAI (2015)Google Scholar
  28. 28.
    Yin, M., Chen, Y., Sun, Y.A.: The effects of performance-contingent financial incentives in online labor markets (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Jiacheng Liu
    • 1
  • Feilong Tang
    • 2
    Email author
  • Long Chen
    • 1
  • Liang Qiao
    • 1
  • Yanqin Yang
    • 3
    • 4
  • Wenchao Xu
    • 3
    • 4
  1. 1.School of SoftwareShanghai Jiaotong UniversityMinghangChina
  2. 2.Department of Computer Science and EngineeringShanghai Jiaotong UniversityMinghangChina
  3. 3.Department of Computer Science and TechnologyEast China Normal UniversityShanghaiChina
  4. 4.Shanghai Key Laboratory of Multidimensional Information ProcessingEast China Normal UniversityShanghaiChina

Personalised recommendations