Active Image Clustering with Pairwise Constraints from Humans



We propose a method of clustering images that combines algorithmic and human input. An algorithm provides us with pairwise image similarities. We then actively obtain selected, more accurate pairwise similarities from humans. A novel method is developed to choose the most useful pairs to show a person, obtaining constraints that improve clustering. In a clustering assignment, elements in each data pair are either in the same cluster or in different clusters. We simulate inverting these pairwise relations and see how that affects the overall clustering. We choose a pair that maximizes the expected change in the clustering. The proposed algorithm has high time complexity, so we also propose a version of this algorithm that is much faster and exactly replicates our original algorithm. We further improve run-time by adding two heuristics, and show that these do not significantly impact the effectiveness of our method. We have run experiments in three different domains, namely leaf, face and scene images, and show that the proposed method improves clustering performance significantly.


Clustering Active learning Human in the loop  Pairwise constraints Image labeling 


  1. Angluin, D. (1987). Queries and concept learning. Machine Learning, 2(4), 319–342.Google Scholar
  2. Basu, S., Banerjee, A., & Mooney, R. J. (2002). Semi-supervised clustering by seeding. In C. Sammut & A. G. Hoffmann (eds.), ICML (pp. 27–34). San Francisco, CA: Morgan Kaufmann. ISBN 1-55860-873-7.Google Scholar
  3. Basu, S., Banerjee, A., & Mooney, R. J. (2004). Active semi-supervision for pairwise constrained clustering. In Fourth SIAM International Conference on Data Mining. ISBN 0-89871-568-7.Google Scholar
  4. Basu, S., Davidson, I., & Wagstaff, K. (2008). Constrained clustering: Advances in algorithms. Data mining and knowledge discovery series. Washington, DC: IEEE Computer SocietyGoogle Scholar
  5. Biswas, A. & Jacobs, D. (2011). Large scale image clustering with active pairwise constraints. In International Conference in Machine Learning 2011 Workshop on Combining Learning Strategies to Reduce Label Cost.Google Scholar
  6. Biswas, A. & Jacobs, D. W. (2012). Active image clustering: Seeking constraints from humans to complement algorithms. In IEEE Conference on CVPR (pp. 2152–2159).Google Scholar
  7. Branson, S., Perona, P., & Belongie, S. (2011). Strong supervision from weak annotation: Interactive training of deformable part models. In D. N. Metaxas, L. Quan, A. Sanfeliu, & L. J. Van Gool (eds.), IEEE Internaional Conference on Computer Vision (pp. 1832–1839).
  8. Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P., & Belongie, S. (2010). Visual recognition with humans in the loop. In K. Daniilidis, P. Maragos, & Paragios, N. (eds.), ECCV Lecture Notes in Computer Science (Vol. 6314, pp. 438–451). Berlin: Springer. ISBN 978-3-642-15560-4. doi: 10.1007/978-3-642-15561-1.
  9. Chellappa, R., & Jain, A. K. (1993). Markov random fields: Theory and applications. Boston: Academic Press.Google Scholar
  10. Dagli, C. K., Rajaram, S., & Huang, T. S. (2006). Utilizing information theoretic diversity for SVM active learn. In ICPR (pp. 506–511). doi: 10.1109/ICPR.2006.1161.
  11. Davidson, I., & Ravi, S. S. (2009). Using instance-level constraints in agglomerative hierarchical clustering: Theoretical and empirical results. Data Mining and Knowledge Discovery, 18(2), 257–282.CrossRefMathSciNetGoogle Scholar
  12. Felzenszwalb, P. F., McAllester, D., Ramanan, D. (2008). A discriminatively trained, multiscale, deformable part model. In CVPR (pp. 1–8). doi: 10.1109/CVPR.2008.4587597.
  13. Gomes, R., Welinder, P., Krause, A., & Perona, P. (2011). Crowdclustering. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira, & K. Q. Weinberger (eds.), NIPS (pp. 558–566).
  14. Grira, N., Crucianu, M., & Boujemaa, N. (2005). Active semi-supervised fuzzy clustering for image database categorization. In ACM SIGMM International Workshop on Multimedia Information Retrieval (pp. 9–16).Google Scholar
  15. Guo, Y. & Greiner, R. (2007). Optimistic active-learning using mutual information. In M. M. Veloso (ed.), IJCAI (pp. 823–829).
  16. Holub, A. D., Perona, P., & Burl, M. C. (2008). Entropy-based active learning for object recognition. In Online Learning for Classification Workshop (pp. 1–8). doi: 10.1109/CVPRW.2008.4563068.
  17. Huang, R., & Lam, W. (2009). An active learning framework for semi-supervised document clustering with language modeling. Data & Knowledge Engineering, 68(1), 49–67.CrossRefGoogle Scholar
  18. Huang, R., Wai, L. & Zhigang, Z. (2007). Active learning of constraints for semi-supervised text clustering. In SDM, SIAM.Google Scholar
  19. Jain, A. K. (2010). Data clustering: 50 years beyond K-means. Pattern Recognition Letters, 31(8), 651–666. doi: 10.1016/j.patrec.2009.09.011.CrossRefGoogle Scholar
  20. Jain, P. & A. Kapoor. (2009). Active learning for large multi-class problems. In CVPR (pp. 762–769). doi: 10.1109/CVPRW.2009.5206651.
  21. Joshi, A. J., Porikli, F., & Papanikolopoulos, N. (2010). Breaking the interactive bottleneck in multi-class classification with active selection and binary feedback. In IEEE Conference on CVPR (pp. 2995–3002). doi: 10.1109/CVPR.2010.5540047.
  22. Kapoor, A., Horvitz, E., Basu, S. (2007). Selective supervision: Guiding supervised learning with decision-theoretic active learning. In M. M. Veloso (ed.), IJCAI (pp. 877–882).
  23. Kruskal, J. (1956). On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the AMS, 2, 48–50.CrossRefMathSciNetGoogle Scholar
  24. Kumar, N., Belhumeur, P. N., Biswas, A., Jacobs, D. W., Kress, W. J., Lopez, I. C., & Soares, J. V. B. (2012) Leafsnap: A computer vision system for automatic plant species identification. In ECCV 2, Lecture Notes in Computer Science (Vol. 7573, pp. 502–516).Google Scholar
  25. Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009). Attribute and simile classifiers for face verification. In IEEE Conference on ICCV (pp. 365–372).Google Scholar
  26. MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematics Statistics and Probability. Google Scholar
  27. Mallapragada, P. K., Jin, R., & Jain, A. K. (2008). Active query selection for semi-supervised clustering. In ICPR (pp. 1–4).Google Scholar
  28. Martinez, A. M., & Benavente, R. (1998). The AR face database. CVC Technical, Report #24.Google Scholar
  29. Punera, K., & Ghosh, J. (2008). Consensus-based ensembles of soft clusterings. Applied Artificial Intelligence, 22(7 &8), 780–810.CrossRefGoogle Scholar
  30. Quattoni, A. & Torralba, A. (2009). Recognizing indoor scenes. In CVPR (pp. 413–420). doi: 10.1109/CVPRW.2009.5206537.
  31. Settles, B. (2010). Active learning literature survey. Technical report.Google Scholar
  32. Siddiquie, B., & Gupta, A. (2010). Beyond active noun tagging: Modeling contextual interactions for multi-class active learning. In IEEE Conference on CVPR (pp. 2979–2986). doi: 10.1109/CVPR.2010.5540044.
  33. Sugar, C. A., & James, G. M. (2003). Finding the number of clusters in a data set: An information theoretic approach. Journal of the American Statistical Association, 98, 750–763.CrossRefMATHMathSciNetGoogle Scholar
  34. Tan, P.-N., Steinbach, M., & Kumar, V. (2005). Introduction to Data Mining. Upper Saddle River, NJ: Addison-Wesley. ISBN 0-321-32136-7.Google Scholar
  35. Vijayanarasimhan, S., & Grauman, K. (2011). Large-scale live active learning: Training object detectors with crawled data and crowds. In IEEE conference on CVPR (pp. 1449–1456). doi: 10.1109/CVPR.2011.5995430.
  36. Vijayanarasimhan, S. & Grauman, K. (2012). Active frame selection for label propagation in videos. In A. W. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, & C. Schmid (eds.), ECCV 5, Lecture Notes in Computer Science (Vol. 7576, pp. 496–509). Berlin: Springer. ISBN 978-3-642-33714-7.Google Scholar
  37. Vijayanarasimhan, S., Jain, P., & Grauman, K. (2010). Far-sighted active learning on a budget for image and video recognition. In IEEE Conference on CVPR (pp. 3035–3042). doi: 10.1109/CVPR.2010.5540055.
  38. Wagstaff, K., Cardie, C., Rogers, S., & Schroedl, S. (2001). Constrained K-means clustering with background knowledge. In Proceedings of 18th International Conference on Machine Learning (pp. 577–584). San Francisco, CA: Morgan Kaufmann.Google Scholar
  39. Wang, X., & Davidson, I. (2010). Active spectral clustering. In ICDM (pp. 561–568). IEEE Computer Society.Google Scholar
  40. Xiong, C., Johnson, D. & Corso, J. J. (2012). Spectral active clustering via purification of the \(k\)-nearest neighbor graph. In ECDM.Google Scholar
  41. Xu, Q., desJardins, M., & Wagstaff, K. (2005). Active constrained clustering by examining spectral eigenvectors. In Discovery, science (Vol. 3735). Berlin: Springer.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Computer Science DepartmentUniversity of MarylandCollege ParkUSA

Personalised recommendations