Abstract
Data clustering is a popular approach for automatically finding classes, concepts, or groups of patterns. In practice, this discovery process should avoid redundancies with existing knowledge about class structures or groupings, and reveal novel, previously unknown aspects of the data. In order to deal with this problem, we present an extension of the information bottleneck framework, called coordinated conditional information bottleneck, which takes negative relevance information into account by maximizing a conditional mutual information score subject to constraints. Algorithmically, one can apply an alternating optimization scheme that can be used in conjunction with different types of numeric and non-numeric attributes. We discuss extensions of the technique to the tasks of semi-supervised classification and enumeration of successive non-redundant clusterings. We present experimental results for applications in text mining and computer vision.
Similar content being viewed by others
References
Bucila C, Gehrke J, Kifer D, White W (2002) Dualminer: A dual-pruning algorithm for itemsets with constraints. In: Proceedings of the Eighth ACM SIGKDD international conference on knowledge discovery and data mining, pp 241–272
Chechik G, Tishby N (2002) Extracting relevant structures with side information. Adv Neural Inf Process Syst 15:857–864
Craven M, DiPasquo D, Freitag D, McCallum AK, Mitchell TM, Nigam K, Slattery S (1998) Learning to extract symbolic knowledge from the World Wide Web. In: Proceedings of the 15th conference on artificial intelligence, pp 509–516
Friedman N, Mosenzon O, Slonim N, Tishby N (2001) Multivariate information bottleneck. In: Proceedings of the 17th conference on uncertainty in artificial intelligence, pp 152–161
Gondek D, Hofmann T (2003) Conditional information bottleneck clustering. In: Proceedings of the 3rd IEEE international conference on data mining, workshop on clustering large data sets
Klein D, Kamvar SD, Manning CD (2002) From instance-level constraints to space-level constraints: Making the most of prior knowledge in data clustering. In: Proceedings of the 19th international conference on machine learning, pp 307–314
McCullagh P, Nelder J (1989) Generalized linear models. Chapman & Hall, London, UK
Ng RT, Lakshmanan LV, Han J, Pang A (1998) Exploratory mining and pruning optimizations of constrained association rule. In: Proceedings of the 1998 ACM SIGMOD international conference on management of data, pp 13–24
Nigam K, McCallum AK, Thrun S, Mitchell TM (2000) Text classification from labeled and unlabeled documents using EM. Mach Learn 39(2–3):103–134
Rose K (1998) Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proc IEEE 80:2210–2239
Slonim N, Friedman N, Tishby N (2002) Unsupervised document classification using sequential information maximization. In: Proceedings of the 25th ACM SIGIR international conference on research and development in information retrieval, pp 129–136
Tishby N, Pereira FC, Bialek W (1999) The information bottleneck method. In: Proceedings of the 37th annual allerton conference on communication, control and computing, pp 368–377
Tung A, Ng R, Han J, Lakshmanan L (2001) Constraint-based clustering in large databases. In: Proceedings of the 8th international conference on database theory, pp 405–419
Wagstaff K, Cardie C (2000) Clustering with instance-level constraints. In: Proceedings of 17th international conference on machine learning, pp 1103–1110
Xing EP, Ng AY, Jordan MI, Russell S (2003) Distance metric learning, with application to clustering with side-information. Adv Neural Inf Process Syst 15:505–512
Zhong S, Ghosh J (2003) Model-based clustering with soft balancing. In: Proceedings of the 3rd IEEE international conference on data mining, pp 459–466
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Gondek, D., Hofmann, T. Non-redundant data clustering. Knowl Inf Syst 12, 1–24 (2007). https://doi.org/10.1007/s10115-006-0009-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-006-0009-7