Hashing Supported Iterative MapReduce Based Scalable SBE Reduct Computation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10722)

Abstract

Feature Selection plays a major role in preprocessing stage of Data mining and helps in model construction by recognizing relevant features. Rough Sets has emerged in recent years as an important paradigm for feature selection i.e. finding Reduct of conditional attributes in given data set. Two control strategies for Reduct Computation are Sequential Forward Selection (SFS), Sequential Backward Elimination(SBE). With the objective of scalable feature seletion, several MapReduce based approaches were proposed in literature. All these approaches are SFS based and results in super set of reduct i.e. with redundant attributes. Even though SBE approaches results in exact Reduct, it requires lot of data movement in shuffle and sort phase of MapReduce. To overcome this problem and to optimize the network bandwidth utilization, a novel hashing supported SBE Reduct algorithm(MRSBER_Hash) is proposed in this work and implemented using Iterative MapReduce framework of Apache Spark. Experiments conducted on large benchmark decision systems have empirically established the relevance of proposed approach for decision systems with large cardinality of conditional attributes.

Keywords

Rough Sets Reduct Iterative MapReduce Apache Spark Scalable feature selection 

References

  1. 1.
    Baadal: the iitd computing cloud (2011). http://www.cc.iitd.ernet.in
  2. 2.
    Dataset used for experiments (1999). http://kdd.ics.uci.edu/databases/kddcup99/
  3. 3.
    Uci machine learning repository. https://archive.ics.uci.edu/ml/datasets (2013)
  4. 4.
    Ekanayake, J., Li, H., Zhang, B., Gunarathne, T., Bae, S.-H., Qiu, J., Fox, G.C.: Twister: a runtime for iterative mapreduce. In: HPDC, pp. 810–818. ACM (2010)Google Scholar
  5. 5.
    Pawlak, Z.: Rough sets. Int. J. Parallel Program. 11(5), 341–356 (1982)MATHGoogle Scholar
  6. 6.
    P.S.V.S., S.P., Raghavendra Rao, C.: Extensions to IQuickReduct. In: Sombattheera, C., Agarwal, A., Udgata, S.K., Lavangnananda, K. (eds.) MIWAI 2011. LNCS (LNAI), vol. 7080, pp. 351–362. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-25725-4_31 CrossRefGoogle Scholar
  7. 7.
    Shen, Q., Jensen, R.: Rough set-based feature selection: a review. In: Rough Computing: Theories, Technologies and Applications, pp. 70–107. IGI Global (2008)Google Scholar
  8. 8.
    Sai Prasad, P.S.V.S., Bala Subrahmanyam, H., Singh, P.K.: Scalable IQRA_IG algorithm: an iterative MapReduce approach for reduct computation. In: Krishnan, P., Radha Krishna, P., Parida, L. (eds.) ICDCIT 2017. LNCS, vol. 10109, pp. 58–69. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-50472-8_5 CrossRefGoogle Scholar
  9. 9.
    Singh, P.K., Sai Prasad, P.S.V.S.: Scalable quick reduct algorithm: iterative mapreduce approach. In: CODS, pp. 25:1–25:2. ACM (2016)Google Scholar
  10. 10.
    Yang, Y., Chen, Z., Liang, Z., Wang, G.: Attribute reduction for massive data based on rough set theory and MapReduce. In: Yu, J., Greco, S., Lingras, P., Wang, G., Skowron, A. (eds.) RSKT 2010. LNCS (LNAI), vol. 6401, pp. 672–678. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-16248-0_91 CrossRefGoogle Scholar
  11. 11.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: Cluster computing with working sets. In: HotCloud. USENIX Association (2010)Google Scholar
  12. 12.
    Zhang, J., Li, T., Pan, Y.: Parallel large-scale attribute reduction on cloud systems. CoRR, abs/1610.01807 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Quadratic Insights Pvt Ltd.HyderabadIndia
  2. 2.School of Computer and Information SciencesUniversity of HyderabadHyderabadIndia

Personalised recommendations