Web Scale Competitor Discovery Using Mutual Information

  • Rui Li
  • Shenghua Bao
  • Jin Wang
  • Yuanjie Liu
  • Yong Yu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4093)


The web with its rapid expansion has become an excellent resource for gathering information and people’s opinion. A company owner wants to know who is the competitor, and a customer also wants to know which company provides similar product or service to what he/she is in want of. This paper proposes an approach based on mutual information, which focuses on mining competitors of the entity(such as company, product, person ) from the web. The proposed techniques first extract a set of candidates of the input entity, and then rank them according to the comparability, and finally find and organize the reviews related to both original entity and its competitors. A novel system called ”CoDis” based upon these techniques is implemented, which is able to automate the tedious process in a domain-independent and web-scale dynamical manner. In the experiment we use 32 different entities distributed in varied domains as inputs and the CoDis discovers 143 competitors. The experimental results show that the proposed techniques are highly effective.


Search Engine Mutual Information Brand Product Domain Information Football Club 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Zanasi, A.: Information gathering and analysis competitive intelligence through data mining public sources. Competitive Intelligence Review 9(1), 44–54Google Scholar
  2. 2.
    Morinaga, S., Yamanishi, K., Tateishi, K., Fukushinna, T.: Mining product reputations on the web. kdd-02, 2002. In: KDD 2002 (2002)Google Scholar
  3. 3.
    Hu, M., Liu, B.: Mining and summarizing customer reviews. In: KDD 2004 (2004)Google Scholar
  4. 4.
    Califf, M., Mooney, R.: Graph drawing by force-directed placement. Journal of Machine Learing Research 4, 117–210 (2003)Google Scholar
  5. 5.
    Freitag, D., McCallum., A.: Information extraction with hmms and shrinkage. In: Processdings of the AAAI-99 Workshop on Machine Learning for Information Extraction (1999)Google Scholar
  6. 6.
    Ashish, N., Knoblock, C.: Wrapper generation for semi-strcutured internet sources. SIGMOD Record 26(4) (1996)Google Scholar
  7. 7.
    Cohen, W., Hurst, M., Jensen, L.: A flexible learning system for wrapping tables and lists in html documents. In: Proceedings of World Wide Web (WWW 2002) (2002)Google Scholar
  8. 8.
    Hearst, M.: Automatic acquisition of hyponyms from large text corpora. In: Proceedings of the 14th International Conference on Computational Linguistics (1992)Google Scholar
  9. 9.
    Charniak, E., Berland, M.: Finding parts in very large corpora. In: Proceeding of the 37th Annual Meeting of the ACL (1999)Google Scholar
  10. 10.
    Hahn, U., Schnattinger, K.: Towards text knowledge engineering. In: Proceedings of the 15th Naional Conference on Artificial Intelligence and the 10th Conference on Innovative Application of Artificial Intelligence (AAAI 1998/IAAI 1998), pp. 524–531 (1998)Google Scholar
  11. 11.
    Cimino, P., Handschuh, S., Staab, S.: Towards the self-annotating web. In: Processdings of World Wide Web (WWW 2004) (2004)Google Scholar
  12. 12.
    Etzioni, O., Cafarella, M., Downey, D., Kok, S., Popcscu, A., Shaked, T., Soderland, S., Weld, S.: Web-scale information extraction in knowitall(preliminary results). In: Processdings of World Wide Web (WWW 2004) (2004)Google Scholar
  13. 13.
    Liu, B., Chin, C.: Mining topic-specific concepts and definitions on the web. In: Processdings of World Wide Web (WWW 2003) (2003)Google Scholar
  14. 14.
    Turney: Mining the web for synonyms:pmi-ir versus lsa on toefl. In: Proceedings of the Twellfth European Conference of MachineLearning (2001)Google Scholar
  15. 15.
    Zhu, J., Goncalves, A.L., Uren, V.S., Motta, E., Parcheco, R.: Mining web data for competency management. In: Proc. of 2005 International Conference on Web Intelligence (2005)Google Scholar
  16. 16.
    Zhu, G.J.: Corder:comunity relation discovery by named entity recognition. In: K-CAP (2005)Google Scholar
  17. 17.
    Etzioni, O.: Moving up the information food chain: Softbots as information carnivores. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence (1996)Google Scholar
  18. 18.
    Xu, J., H., Y.: Using svm to extract acronyms in text. In: Proceedings of International Conference on Machine Learning and Cybernetics (ICMLC 2005) (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Rui Li
    • 1
  • Shenghua Bao
    • 1
  • Jin Wang
    • 1
  • Yuanjie Liu
    • 1
  • Yong Yu
    • 1
  1. 1.Department of Computer Science and EngineeringShanghai JiaoTong UniversityShanghaiP.R. China

Personalised recommendations