Mining Associations Between Two Categories Using Unstructured Text Data in Cloud

  • Yanqing Ji
  • Yun Tian
  • Fangyang Shen
  • John Tran
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 738)


Finding associations between itemsets within two categories (e.g., drugs and adverse effects, genes and diseases) are very important in many domains. However, these association mining tasks often involve computation-intensive algorithms and a large amount of data. This paper investigates how to leverage MapReduce to effectively mine the associations between itemsets within two categories using a large set of unstructured data. While existing MapReduce-based association mining algorithms focus on frequent itemset mining (i.e., finding itemsets whose frequencies are higher than a threshold), we proposed a MapReduce algorithm that could be used to compute all the interestingness measures defined on the basis of a 2 × 2 contingency table. The algorithm was applied to mine the associations between drugs and diseases using 33,959 full-text biomedical articles on the Amazon Elastic MapReduce (EMR) platform. Experiment results indicate that the proposed algorithm exhibits linear scalability.


Association mining MapReduce Cloud computing 


  1. 1.
    L. Geng, H.J. Hamilton, Interestingness measures for data mining: a survey, ACM. Comput. Surv. 38, (2006)Google Scholar
  2. 2.
    P.-N. Tan, M. Steinbach, V. Kumar, Introduction to Data Mining, 1st edn. (Addison Wesley, Boston, 2005)Google Scholar
  3. 3.
    R. Agrawal, R. Srikant, Fast algorithms for mining association rules. Presented at the proceedings of the 20th international conference on very large databases, Santiago, Chile, 1994Google Scholar
  4. 4.
    J. Han, J. Pei, Y. Yin, Mining frequent patterns without candidate generation. SIGMOD. Rec. 29, 1–12 (2000)CrossRefGoogle Scholar
  5. 5.
    N.C.f.B. Information, PubMed, 2017.
  6. 6.
    J. Dean, S. Ghemawat, MapReduce: simplified data processing on large clusters. Commun. ACM. 51, 107–113 (2008)CrossRefGoogle Scholar
  7. 7.
    F. Kovács, J. Illés, Frequent itemset mining on hadoop, in 2013 IEEE 9th International Conference on Computational Cybernetics (ICCC), 2013, pp. 241–245Google Scholar
  8. 8.
    X.Y. Yang, Z. Liu, Y. Fu, MapReduce as a programming model for association rules algorithm on Hadoop, in The 3rd International Conference on Information Sciences and Interaction Sciences, 2010, pp. 99–102Google Scholar
  9. 9.
    K. Chavan, P. Kulkarni, P. Ghodekar, S.N. Patil, Frequent itemset mining for Big data, in 2015 International Conference On Green Computing and Internet of Things (ICGCIoT), 2015, pp. 1365–1368Google Scholar
  10. 10.
    N. Li, L. Zeng, Q. He, Z. Shi, Parallel implementation of apriori algorithm based on MapReduce, in 2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, 2012, pp. 236–241Google Scholar
  11. 11.
    C. Doulkeridis, K. Nørvåg, A survey of large-scale analytical query processing in MapReduce. VLDB J. 23, 355–380 (2014)CrossRefGoogle Scholar
  12. 12.
    Y. Ji, Y. Tian, F. Shen, J. Tran, Leveraging MapReduce to efficiently extract associations between biomedical concepts from large text data. Microprocess. Microsyst. 46, 202–210 (2016)CrossRefGoogle Scholar
  13. 13.
    Y. Ji, Y. Tian, F. Shen, J. Tran, High-performance biomedical association mining with MapReduce, in 2015 12th International Conference on Information Technology—New Generations, 2015, pp. 465–470Google Scholar
  14. 14.
    T.R. Conference, TREC 2006 Genomics Track.
  15. 15.
    A.R. Aronson, F.M. Lang, An overview of MetaMap: historical perspective and recent advances. J. Am. Med. Inform. Assoc. 17, 229–236 (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Yanqing Ji
    • 1
  • Yun Tian
    • 2
  • Fangyang Shen
    • 3
  • John Tran
    • 4
  1. 1.Department of Electrical and Computer EngineeringGonzaga UniversitySpokaneUSA
  2. 2.Department of Computer ScienceEastern Washington UniversityCheneyUSA
  3. 3.Department of Computer System TechnologyNew York City College of TechnologyBrooklynUSA
  4. 4.Frontier Behavioral HealthSpokaneUSA

Personalised recommendations