Inference of Co-occurring Classes: Multi-class and Multi-label Classification

  • Tal Sobol-Shikler
Part of the Studies in Computational Intelligence book series (SCI, volume 386)


The inference of co-occurring classes, i.e. multi-class and multi-label classification, is relevant to various aspects of human cognition, human-machine interactions and to the analysis of knowledge domains and processes that have traditionally been investigated in the social sciences, life sciences and humanities. Human knowledge representations usually comprise multiple classes which are rarely mutually exclusive. Each instance (sample) can belong to one or more of these classes. However, full labeling is not always possible, and the size of the consistently labeled is often limited. The level of existence of a class often varies between instances or sub-classes. The features that distinguish the classes are not always known, and can be different between classes. Hence, methods should be devised to perform multi-class and multi-label classification, and to approach the challenges entailed in the complex knowledge domains. This chapter surveys current approaches to multi-class and multi-label classification in various knowledge domains, and approaches to data annotation (labeling). In particular, it presents a classification algorithm designed for inferring the levels of co-occurring affective states (emotions, mental states, attitudes etc.) from their non-verbal expressions in speech.


Association Rule Knowledge Domain Affective State Binary Classifier Image Annotation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, C.C.: Data streams: An overview and scientific applications. In: Gaber, M.M. (ed.) Scientific Data Mining and Knowledge Discovery: Principles and Foundations. Springer, Heidelberg (2010)Google Scholar
  2. 2.
    Allwein, E.L., Schapire, R., Singer, Y.: Reducing multiclass to binary: a unifying approach for margin classiers. J. of Machine Learning Research 1, 113–141 (2001)zbMATHMathSciNetGoogle Scholar
  3. 3.
    Al-Naymat, G.: Data mining and discovery of astronomical knowledge. In: Gaber, M.M. (ed.) Scientific data mining and knowledge discovery: Principles and foundations. Springer, Heidelberg (2010)Google Scholar
  4. 4.
    Amit, Y., Dekel, O., Singer, Y.: A boosting algorithm for label covering in multilabel problems. In: Proc. AISTATS (2007)Google Scholar
  5. 5.
    Barutcuoglu, Z., Schapire, R.E., Troyanskaya, O.G.: Hierarchical multi-label prediction of gene function. Bioinformatics 22(7), 830–836 (2006)CrossRefGoogle Scholar
  6. 6.
    Baron-Cohen, S., Golan, O., Wheelwright, S., Hill, J.J.: Mindreading: The interactive guide to emotions. Jessica Kingsley Limited London (2004),
  7. 7.
    Bengio, S., Pereira, F., Singer, Y., Strelow, D.: Group Sparse Coding (2009)Google Scholar
  8. 8.
    Boutell, M.R., Luo, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Pattern Recognition 37(9), 1757–1771 (2004)CrossRefGoogle Scholar
  9. 9.
    Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. The methods of paired comparisons. Biometrika 39, 324–345 (1952)zbMATHMathSciNetGoogle Scholar
  10. 10.
    Bredensteiner, E.J., Bennett, K.P.: Multicategory classification by support vector machines. Computational Optimization and Applications 12, 53–79 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Chen, M.Y., Christel, M., Hauptmann, A., Wactlar, H.: Putting active learning into multimedia applications: dynamic definition and refinement of concept classifiers. In: Proc. of the 13th Annual ACM International Conference on Multimedia, MULTIMEDIA 2005, pp. 902–911. ACM Press, New York (2005)CrossRefGoogle Scholar
  12. 12.
    Crammer, K., Singer, Y.: On the algorithmic implementation of multi-class kernel-based vector machines. J. Machine Learning Research 2, 265–292 (2001)Google Scholar
  13. 13.
    Dekel, O., Manning, C.D., Singer, Y.: Log linear models for label ranking (2003)Google Scholar
  14. 14.
    Dekel, O., Shamir, O.: Multiclass-multilabel learning when the label set grows with the number of examples. Technical Report MSR-TR-2009-163, Microsoft Research multi-label (2009)Google Scholar
  15. 15.
    Douglas-Cowie, E., Campbell, N., Cowie, R., Roach, P.: Emotional speech: towards a new generation of databases. Speech Communication 40, 33–60 (2003)CrossRefzbMATHGoogle Scholar
  16. 16.
    Ekman, P.: Basic emotion. In: Power, M., Dalgleish, T. (eds.) Handbook of cognition and emotion, Wiley, Chihester (1999)Google Scholar
  17. 17.
    el Kaliouby, R., Robinson, P.: Real-time inference of complex mental states from facial expressions and head gestures. In: Real-time vision for HCI, pp. 181–200. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  18. 18.
    Feng, S., Xu, D.: Transductive multi-instance multi-label learning algorithm with application to automatic image annotation. Expert Systems with Applications 37, 661–670 (2010)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Fernándeza, A., Calderónb, M., Barrenecheab, E., Bustinceb, H., Herrera, F.: Solving multi-class problems with linguistic fuzzy rule based classification systems based on pairwise learning and preference relations. Fuzzy Sets and Systems 161, 3064–3080 (2010)CrossRefMathSciNetGoogle Scholar
  20. 20.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 (1997)Google Scholar
  21. 21.
    Friedman, J.: Another approach to polychotomous classication. Technical report, Department of Statistics, Stanford University (1996),
  22. 22.
    Grimm, S.: Knowledge representation and ontologies. In: Gaber, M.M. (ed.) Scientific data mining and knowledge discovery: Principles and foundations. Springer, Heidelberg (2010)Google Scholar
  23. 23.
    Ghamrawi, N., McCallum, A.: Collective multi-label classification. In: Proc CIKM 2005, Bremen, Germany (2005)Google Scholar
  24. 24.
    Gruber, T.R.: J. Knowledge Acquisition 6(2), 199–221 (1993)CrossRefGoogle Scholar
  25. 25.
    Grundland, M., Dodgson, N.A.: Color search and replace. In: Computational Aesthetics, EUROGRAPHICS, Girona, Spain, pp. 101–109 (2005)Google Scholar
  26. 26.
    Hastie, T., Tibshirani, R.: Classication by pairwise coupling. The Annals of Statistics 26(1), 451–471 (1998)zbMATHMathSciNetGoogle Scholar
  27. 27.
    Hsu, C.W., Lin, C.J.: A comparison of methods for multiclass support vector machines. IEEE Transactions on Neural Networks 13(2), 415–425 (2002)CrossRefGoogle Scholar
  28. 28.
    Hsu, D., Kakade, S., Langford, J., Zhang, T.: Multi-label prediction via compressed sensing. In: NIPS (2009)Google Scholar
  29. 29.
    Hu, J., Lam, K.M., Qiu, G.: A hierarchical algorithm for image multi-labeling. In: Proc. IEEE 17th International Conference on Image Processing, Hong Kong (2010)Google Scholar
  30. 30.
    Knerr, S., Personnaz, L., Dreyfus, G.: Single-layer learning revisited: a stepwise procedure for building and training a neural network. In: Fogelman, J. (ed.) Neurocomputing: Algorithms, Architectures and Applications, Springer, Heidelberg (1990)Google Scholar
  31. 31.
    Laurier, C., Meyers, O., Serra, J., Blech, M., Herrera, P.: Music mood annotator design and integration. In: Proc. 7th International Workshop on Content-Based Multimedia Indexing, pp. 156–161 (2009)Google Scholar
  32. 32.
    Lee, Y., Lin, Y., Wahba, G.: Multi category support vector machines: theory and application to the classification of micro array data and satellite radiance data. J. American Statistical Association 99, 67–81 (2004)CrossRefzbMATHMathSciNetGoogle Scholar
  33. 33.
    Lellmann, J., Becker, F., Schnorr, C.: Convex optimization for multi-class image labeling with a novel family of total variation based regularizers. In: Proc. IEEE 12th International Conference on Computer Vision, ICCV (2009)Google Scholar
  34. 34.
    Li, W., Han, J., Pei, J.: CMAR: Accurate and efficient classification based on multiple class association rule. In: Proc ICDM 2001, San Jose, CA, pp. 369–376 (2001)Google Scholar
  35. 35.
    Li, Y., Tian, Y., Duan, L.Y., Yang, J., Huang, T., Gao, W.: Sequence multi-labeling: a unifiedvideo annotation scheme with spatial and temporal context. IEEE Trans. Multimedia 12(8), 814–828 (2010)CrossRefGoogle Scholar
  36. 36.
    Liu, B., Hsu, H., Ma, Y.: Integrating Classification and association rule mining. In: Proc. KDD 1998, New York (1998)Google Scholar
  37. 37.
    Liu, J., Luo, J., Shah, M.: Recognizing realistic actions from videos in the wild. In: Proc. CVPR (2009)Google Scholar
  38. 38.
    Liu, J., Li, M., et al.: Image annotation via graph learning. Pattern Recognition 42, 218–228 (2009)CrossRefzbMATHGoogle Scholar
  39. 39.
    Lukashevich, H., Abeßer, J., Dittmar, C., Grossmann, H.: From multi-labeling to multi-domain-labeling: A novel two-dimensional approach to music genre classification. In: Proc. 10th International Society for Music Information Retrieval Conference (ISMIR 2009), pp. 459–464 (2009)Google Scholar
  40. 40.
    Malkevitch, J.: The process of electing a president. AMS, American Mathematical Society (April 2008),
  41. 41.
    Marsland, S.: Machine learning: an algorithmic perspective. Chapman & Hall/CRC Machine learning & pattern recognition series, FL, USA (2009)Google Scholar
  42. 42.
    McCallum, A.: Multi-label text classification with a mixture model trained by EM. In: Proc AAAI 1999 Workshop on Text Learning (1999)Google Scholar
  43. 43.
    Montejo-Raez, A., Urena-Lopez, L.A.: Binary classifiers versus AdaBoost for labeling of digital documentsGoogle Scholar
  44. 44.
    Montejo-R´aez, A., Steinberger, R., Ure˜na-L´opez L.A.: Adaptive selection of base classifiers in one-against all learning for large multi-labeled collections, vol. (3230), pp. 1–12 (2004)Google Scholar
  45. 45.
    Polikar, R.: Ensemble based systems in decision making. IEEE circuits and systems magazinee (3rd quarter) (2006)Google Scholar
  46. 46.
    Peng, W., Gero, J.S.: Concept formation in scientific knowledge discovery. In: Gaber, M.M. (ed.) Scientific Data Mining and Knowledge Discovery: Principles and foundations, Springer, Heidelberg (2010)Google Scholar
  47. 47.
    Qi, G.J., et al.: Correlative multi-label video annotation. In: Proc. MM 2007, Bavaria, Germany, pp. 17–26 (2007)Google Scholar
  48. 48.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)Google Scholar
  49. 49.
    Rak, R., Kurgan, L., Reformat, M.: Multi-label Associative Classification of Medical Documents from MEDLINE. In: Proc. 4th International Conference on Machine Learning and Applications, ICMLA 2005 (2005)Google Scholar
  50. 50.
    Sahli, N., Jabeur, N.: Knowledge discovery and reasoning in geospatial applications. In: Gaber, M.M. (ed.) Scientific data mining and knowledge discovery: Principles and foundations. Springer, Heidelberg (2010)Google Scholar
  51. 51.
    Scaringella, N., Zoia, G., Mlynek, D.: Automatic genre classification of music content: a survey. IEEE Signal Processing Magazine 23, 133–141 (2006)CrossRefGoogle Scholar
  52. 52.
    Schapire, R.E.: The strength of weak learnability. Machine Learning 5(2), 197–227 (1990)Google Scholar
  53. 53.
    Schapire, R., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Machine Learning 37, 297–336 (1999)CrossRefzbMATHGoogle Scholar
  54. 54.
    Schapire, R., Singer, Y.: BoosTexter: A boosting-based system for text categorization. Machine Learning 39(2/3), 135–168 (2000)CrossRefzbMATHGoogle Scholar
  55. 55.
    Shiraishi, Y., Fukumizu, K.: Statistical approaches to combining binary classifiers for multi-class classification. Neurocomputing 74, 680–688 (2011)CrossRefGoogle Scholar
  56. 56.
    Singh, M., Curran, E., Cunningham, P.: Active Learning for Multi-Label Image Annotation. Technical Report UCD-CSI-2009-01, University College Dublin (2009)Google Scholar
  57. 57.
    Sobol-Shikler, T.: Automatic Inference of Complex Affective States. Computer Speech and Language 25, 45–62 (2011); doi:10.1016/j.csl.2009.12.005CrossRefGoogle Scholar
  58. 58.
    Sobol-Shikler, T.: Analysis of affective expressions in speech, Tech. report, University of Cambridge (2009)Google Scholar
  59. 59.
    Sobol-Shikler, T.: Multi-modal analysis of human computer interaction using automatic inference of aural expressions in speech. In: Proc. IEEE International Conference on Systems, Man & Cybernetics (SMC), Singapore (2008)Google Scholar
  60. 60.
    Sobol-Shikler, T., Robinson, P.: Classification of complex information: Inference of co-occurring affective states from their expressions in speech. IEEE Trans. Pattern Analysis and Machine Intelligence 32(7), 1284–1297 (2010); doi:10.1109/TPAMI.2009.107CrossRefGoogle Scholar
  61. 61.
    Sowa, J.F.: Knowledge representation. Brokks Cole Publishing, CA (2000)Google Scholar
  62. 62.
    Tanner, S., Stein, C., Graves, S.J.: On-board data mining. In: Gaber, M.M. (ed.) Scientific Data Mining and Knowledge Discovery: Principles and Foundations. Springer, Heidelberg (2010)Google Scholar
  63. 63.
    Thabtah, F.A., Cowling, P., Peng, Y.: MMAC: A new multi-class, multi-label associative classification approach. In: Proc. 4th IEEE International Conference on Data Mining, ICDM 2004(2004)Google Scholar
  64. 64.
    Tsoumakas, G., Katakis, I., Vlahavas, I. (20??) Random k-labelsets for multi-label classification. IEEE Trans. Knowledge and Data Engineering (2010)Google Scholar
  65. 65.
    Vapnik, V.: Estimation of Dependences Based on Empirical Data. Springer, Heidelberg (1982)zbMATHGoogle Scholar
  66. 66.
    Wang, J.Z., Li, J., Wiederhold, G.: Simplicity: Semantics-sensitive integrated matching for picture Libraries. IEEE Trans. Pattern Analysis and Machine Intelligence 23(9), 947–963 (2001)CrossRefGoogle Scholar
  67. 67.
    Wang, H., Huang, M., Wang, X.Z.: A generative probabilistic model for multi-label classification. In: Proc. 8th IEEE International Conference on Data Mining (2008)Google Scholar
  68. 68.
    Wang, M., Zhou, X., Chua, T.S.: Automatic Image Annotation via Local Multi-Label Classification. In: Proc. CIVR 2008, Niagara Falls, Ontario, Canada (2008)Google Scholar
  69. 69.
    Warrell, J., Prince, S.J.D., Moore, A.P.: Epitomized Priors for Multi-labeling ProblemsGoogle Scholar
  70. 70.
    Witten, I.H., Frank, E.: Data mining: practical machine learning tools with java implementations. Morgan Kaufmann, San Francisco (2000)Google Scholar
  71. 71.
    Woitek, P., Brauer, P., Grossmann, H.: A novel tool for capturing conceptualized audio annotations. In: Proc. AM 2010, Pitea, Sweden (2010)Google Scholar
  72. 72.
    Wu, T.F., Lin, C.J., Weng, R.C., Singer, Y.: Probability estimates for multi-class classication by pairwise coupling. Journal of Machine Learning Research 5, 975–1005 (2004)zbMATHGoogle Scholar
  73. 73.
    Yan, R., Yang, J., Hauptmann, A.: Automatically labeling video data using multi-class active learning. In: Proc 9th International Conference on Computer Vision (ICCV 2003), Nice, France, pp. 516–523 (2003)Google Scholar
  74. 74.
    Yang, F., Shi, F., Wang, J.: An Improved GMM-based Method for Supervised Semantic Image Annotation, pp. 506–510 (2009)Google Scholar
  75. 75.
    Yin, X., Han, J.: CPAR: Classification based on predictive association rule. In: Proc. SDM 2003, San Francisco, CA (2003)Google Scholar
  76. 76.
    Zhang, T.: Statistical analysis of some multi-category large margin classification methods. J. Machine Learning Research 5, 1225–1251 (2004)zbMATHGoogle Scholar
  77. 77.
    Zhang, M.L., Zhou, Z.H.: ML-KNN:A lazy learning approach to multi-label learning. Pattern Recognition 40, 2038–2048 (2007)CrossRefzbMATHGoogle Scholar
  78. 78.
    Zhang, T., Liu, S., Xu, C., Lu, H.: Boosted multi-class semi-supervised learning for human action recognition. Pattern recognition (2010); doi:10.1016/j.patcog.2010.06.018Google Scholar
  79. 79.
    Zhu, J., Hastie, T.: Kernel logistic regression and import vector machine. J. Computational and Graphical Statistics 14, 185–205 (2005)CrossRefMathSciNetGoogle Scholar
  80. 80.
    Zhu, J., Rosset, S., Zou, H., Hastie, T.: Multi-class AdaBoost, accessed (February 2011),
  81. 81.
    Zhu, W.: Semantic scene concept learning by an autonomous agent. In: Proc AAAI 2005 (2005)Google Scholar

Copyright information

© Springer Berlin Heidelberg 2012

Authors and Affiliations

  • Tal Sobol-Shikler
    • 1
  1. 1.Ben-Gurion University of the NegevBeer-ShevaIsrael

Personalised recommendations