Word Clouds for Efficient Document Labeling
In text classification the amount and quality of training data is crucial for the performance of the classifier. The generation of training data is done by human labelers - a tedious and time-consuming work. We propose to use condensed representations of text documents instead of the full-text document to reduce the labeling time for single documents. These condensed representations are key sentences and key phrases and can be generated in a fully unsupervised way. The key phrases are presented in a layout similar to a tag cloud. In a user study with 37 participants we evaluated whether document labeling with these condensed representations can be done faster and equally accurate by the human labelers. Our evaluation shows that the users labeled word clouds twice as fast but as accurately as full-text documents. While further investigations for different classification tasks are necessary, this insight could potentially reduce costs for the labeling process of text documents.
KeywordsText classification visualization user interface word clouds document labeling document annotation
Unable to display preview. Download preview PDF.
- 1.Wordle - Beautiful Word Clouds, http://www.wordle.net (accessed: April 25, 2011)
- 2.Baldridge, J., Palmer, A.: How well does active learning actually work?: Time-based evaluation of cost-reduction strategies for language documentation. In: Proc. of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 296–305. Association for Computational Linguistics, Morristown (2009)CrossRefGoogle Scholar
- 3.Druck, G., Mann, G., McCallum, A.: Learning from labeled features using generalized expectation criteria. In: SIGIR 2008: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 595–602. ACM, New York (2008), http://portal.acm.org/citation.cfm Google Scholar
- 6.Guan, H., Zhou, J., Guo, M.: A class-feature-centroid classifier for text categorization. In: Proc. of the International Conference on World Wide Web (WWW), pp. 201–210. ACM, New York (2009)Google Scholar
- 7.Gupta, V., Lehal, G.: A survey of text summarization extractive techniques. Journal of Emerging Technologies in Web Intelligence 2(3) (2010), http://ojs.academypublisher.com/index.php/jetwi/article/view/0203258268
- 10.McCallum, A.K.: Mallet: A machine learning for language toolkit (2002), http://mallet.cs.umass.edu
- 11.Mihalcea, R., Tarau, P.: Textrank: Bringing order into texts. In: Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain (2004), http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf
- 12.Paley, W.B.: TextArc: Showing word frequency and distribution in text. In: Proceedings of IEEE Symposium on Information Visualization, Poster Compendium. IEEE CS Press, Los Alamitos (2002)Google Scholar
- 15.Seifert, C., Kump, B., Kienreich, W., Granitzer, G., Granitzer, M.: On the beauty and usability of tag clouds. In: Proceedings of the 12th International Conference on Information Visualisation (IV), pp. 17–25. IEEE Computer Society, Los Alamitos (2008)Google Scholar
- 16.Settles, B.: Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison (2010), http://pages.cs.wisc.edu/~bsettles/active-learning
- 18.Tomanek, K., Olsson, F.: A web survey on the use of active learning to support annotation of text data. In: Proc. of the NAACL Workshop on Active Learning for Natural Language Processing (HLT), pp. 45–48. Association for Computational Linguistics, Morristown (2009)Google Scholar
- 20.Zhu, X.: Semi-supervised learning literature survey. Tech. Rep. 1530, Computer Sciences, University of Wisconsin (2008), http://pages.cs.wisc.edu/~jerryzhu/pub/ssl_survey.pdf