Abstract
We apply the cross-entropy (CE) method to problems in clustering and vector quantization. The CE algorithm for clustering involves the following iterative steps: (a) generate random clusters according to a specified parametric probability distribution, (b) update the parameters of this distribution according to the Kullback–Leibler cross-entropy. Through various numerical experiments, we demonstrate the high accuracy of the CE algorithm and show that it can generate near-optimal clusters for fairly large data sets. We compare the CE method with well-known clustering and vector quantization methods such as K-means, fuzzy K-means and linear vector quantization, and apply each method to benchmark and image analysis data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
The USC-SIPI Image Database.: http://sipi.usc.edu/services/database/Database.html
TSPLIB : A Traveling Salesman Problem Library. http://www.iwr.uni-heidelberg.de/groups/comopt/software/TSPLIB95/
Ahalt S.C., Krishnamurthy A.K., Chen P., Melton D.E. (1990) Competitive learning algorithms for vector quantization. Neural Net. 3, 277–290
Betke, M., Makris, N.: Fast object recognition in noisy images using simulated annealing. In: Proceedings of the Fifth International Conference on Computer Vision, pp. 523–530 (1995)
Bezdek J. (1981) Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, New York
Botev, Z., Kroese, D.P. Global likelihood optimization via the cross-entropy method, with an application to mixture models. In: Ingalls, R.G., Rossetti, M.D., Smith, J.S., Peters, B.A. (eds.) Proceedings of the 2004 Winter Simulation Conference, IEEE, Washington, DC, December 2004
Brown, L. A survey of image registration techniques. Technical report, Department of Computer Science, Columbia University (1992)
Chen J., Kundu A. (1995) Unsupervised texture segmentation using multichannel decomposition and hidden Markov models. IEEE Trans. Image Process. 4, 603–619
de Boer, P.T., Kroese, D.P., Mannor, S., Rubinstein, R.Y. A tutorial on the cross-entropy method. Ann Oper. Res. Vol. 134, pp.19–67. Springer-Verlag (2005)
Dorigo, M., Di Caro, G. The ant colony optimization meta-heuristic. In: Corne, D., Dorigo, M., Glover, F. (eds.) New Ideas in optimization, pp. 11–32. McGraw-Hill (1999)
Duda R.O., Hart P.E., Stork D.G. (2001) Pattern Classification. Wiley, New York
Geman S., Geman D. (1984) Stochastic relaxation, Gibbs distribution and the Bayesian restoration of images. IEEE Trans. PAMI 6, 721–741
Glover F., Laguna M.L. (1993) Modern Heuristic Techniques for Combinatorial Optimization, Chapter 3: Tabu Search. Blackwell Scientific Publications, Oxford
Goldberg D. (1989) Genetic Algorithms in Search, Optimization and Machine Learning. Addison Wesley, Reading, MA
Hansen P., Mladenović N. (2001) J-means: a new local search heuristic for minimum sum of squares clustering. Pattern Recogn. 34, 405–413
Hansen P., Mladenović N., Perez-Brito D. (2001). Variable neighborhood decomposition search. J Heuristics 7, 335–350
Jain A.K., Murty M.N., Flynn P.J. (1999) Data clustering: a review. ACM Comput. Surv. 31, 264–323
Kaufman L., Rousseeuw P. (1990) Finding Groups in Data, and Introduction to Cluster Analysis. Wiley, New York
Keith, J., Kroese, D.P. Sequence alignment by rare event simulation. In: Proceedings of the 2002 Winter Simulation Conference, pp.~320–327. San Diego (2002)
McLachlan G., Krishnan T. (1997) The EM Algorithm and Extensions. Wiley, New York
Rubinstein R.Y. (1999) The cross-entropy method for combinatorial and continuous optimization. Methodol Comp. Appl. Prob. 2, 127–190
Rubinstein R.Y., Kroese D.P. (2004) The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning. Springer-Verlag, New York
Salomon D. (2000) Data Compression: The Complete Reference. Springer-Verlag, New York
Sherali H.D., Desai J. (2005) A global optimization rlt-based approach for solving the hard clustering problem. J Global Optim. 32, 281–306
Shi L., Olafsson S. (2000) Nested partitioning method for global optimization. Oper. Res. 48(3): 390–407
Spall J.C. (2003) Introduction to Stochastic Search and Optimization. John Wiley, New York
Stork D.G., Yom-Tov E. (2004) Computer Manual to Accompany Pattern Classification. Wiley, New York
Webb A. (1999) Statistical Pattern Recognition. Arnold, London
Yang M.-S., Wu K.-L. (2006) Unsupervised possibilistic clustering. Pattern Recog. 39, 5–21
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kroese, D.P., Rubinstein, R.Y. & Taimre, T. Application of the cross-entropy method to clustering and vector quantization. J Glob Optim 37, 137–157 (2007). https://doi.org/10.1007/s10898-006-9041-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-006-9041-0