Skip to main content
SpringerLink
Log in

Surface Material Recognition Using Active Multi-modal Extreme Learning Machine

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Visual, haptic, and auditory modalities can provide different properties about surface materials and therefore form important perception methods for common material. Nevertheless, how to effectively combine various modalities is an extremely challenging problem. To this end, the active multi-modal framework based on extreme learning machine with multi-scale local receptive fields is developed to fuse the information of different modalities. We conduct multi-modal experiments on the TUM haptic texture database. The experimental results show the highly representative characteristics can be extracted from surface material by the proposed architecture and the three modalities fusion achieves the best performance. The proposed active multi-modal fusion framework shows significant performance improvements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ojala T, Pietikainen M, Maenpaa T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J]. IEEE Trans Pattern Anal Mach Intell 2002;24(7):971–987.

    Article  Google Scholar 

  2. Guo Z, Zhang L, Zhang D. Rotation invariant texture classification using LBP variance (LBPV) with global matching. Pattern Recogn 2010;43(3):706–719.

    Article  Google Scholar 

  3. Soh LK, Tsatsoulis C. Texture analysis of SAR sea ice imagery using gray level co-occurrence matrices[J]. IEEE Trans Geosci Remote Sens 1999;37(2):780–795.

    Article  Google Scholar 

  4. Ruiz L, Fdez-Sarrła A, Recio J. Texture feature extraction for classification of remote sensing data using wavelet decomposition: a comparative study. ISPRS Congress part B 2004;35:1109–1114.

    Google Scholar 

  5. Kim SC, Kang TJ. Texture classification and segmentation using wavelet packet frame and Gaussian mixture model[J]. Pattern Recogn 2007;40(4):1207–1221.

    Article  Google Scholar 

  6. Huang J, Yu ZL, Cai Z, et al. Extreme learning machine with multi-scale local receptive fields for texture classification[J]. Multidim Syst Sign Process. 2016;1–17.

  7. Zheng H, Fang L, Ji M, et al. Deep learning for surface material classification using haptic and visual information[J]. IEEE Trans Multimed 2016;18(12):2407–2416.

    Article  Google Scholar 

  8. Strese M, Schuwerk C, Iepure A, et al. 2017. Multimodal feature-based surface material classification[J]. IEEE Transactions on Haptics.

  9. Corradi T, Hall P, Iravani P. Object recognition combining vision and touch[J]. Robot Biomimet 2017;4 (1):2.

    Article  Google Scholar 

  10. Cao J, Lin Z. 2015. Extreme learning machines on high dimensional and large data applications: a survey. Mathematical Problems in Engineering.

  11. Mayol-Cuevas W u W, Juarez-Guerrero J, Munoz-Gutierrez S. A first approach to tactile texture recognition[C]//. In: 1998 IEEE International Conference on Systems, Man, and Cybernetics, 1998. IEEE; 1998. vol 5, p. 4246–4250.

  12. Cao J, Lin Z, Huang GB, et al. Voting based extreme learning machine. Inform Sci 2012;185(1):66–77.

    Article  Google Scholar 

  13. Cao J, Wang W, Wang J, et al. 2016. Excavation equipment recognition based on novel acoustic statistical features. IEEE Transactions on Cybernetics.

  14. Cao K et al. Classification of uncertain data streams based on extreme learning machine. Cogn Comput 2015; 7(1):150C60.

    Article  Google Scholar 

  15. Liu H, Yu Y, Sun F, et al. Visual-tactile fusion for object recognition[J]. IEEE Trans Autom Sci Eng 2017;14(2):996–1008.

    Article  Google Scholar 

  16. Tang Q, Shen Y, Hu C, et al. Swarm intelligence: based cooperation optimization of multi-modal functions[J]. Cogn Comput 2013;5(1):48–55.

    Article  Google Scholar 

  17. Liu H, Wu Y, Sun F, et al. Weakly paired multimodal fusion for object recognition[J]. IEEE Trans Autom Sci Eng 2018;15(2):784–795.

    Article  Google Scholar 

  18. Liu H, Sun F, Fang B, et al. Multimodal measurements fusion for surface material categorization[J]. IEEE Trans Instrum Measur 2018;67(2):246–256.

    Article  Google Scholar 

  19. Liu Y, Zhang L, Deng P, et al. Common subspace learning via cross-domain extreme learning machine[J]. Cogn Comput 2017;9(3):1–9.

    Google Scholar 

  20. Liu N, Sakamoto JT, Cao J, et al. Ensemble-based risk scoring with extreme learning machine for prediction of adverse cardiac events[J]. Cogn Comput 2017;9(1):1–10.

    Article  Google Scholar 

  21. Huang GB, Bai Z, Kasun LLC, et al. Local receptive fields based extreme learning machine[J]. IEEE Comput Intell Mag 2015;10(2):18–29.

    Article  Google Scholar 

  22. Huang GB, Zhou H, Ding X, Zhang R. Extreme learning machine for regression and multiclass classification. Syst Man Cybern Part B: Cybern IEEE Trans 2012;42(2):513C29.

    Google Scholar 

  23. Yang Y, Wu QMJ. Multilayer extreme learning machine with subnetwork nodes for representation learning. IEEE Trans Cybern 2016;46(11):2570–2583.

    Article  PubMed  Google Scholar 

  24. Wen G, Hou Z, Li H, et al. Ensemble of deep neural networks with probability-based fusion for facial expression recognition[J]. Cogn Comput. 2017;1–14.

  25. Liu N, Sakamoto JT, Cao J, et al. Ensemble-based risk scoring with extreme learning machine for prediction of adverse cardiac events[J]. Cogn Comput. 2017;1–10.

  26. Liu Y, Zhang L, Deng P, et al. Common subspace learning via cross-domain extreme learning machine[J]. Cogn Comput. 2017;1–9.

  27. Kim J, Shin HS, Shin K, Lee M. Robust algorithm for arrhythmia classification in ECG using extreme learning machine. Biomed Eng Online 2009;8:31:1C12.

    Article  Google Scholar 

  28. Menelas B, Hu Y, Lahamy H, et al. Haptic and gesture-based interactions for manipulating geological datasets[C]. In: IEEE International conference on systems, man, and cybernetics. IEEE;2011. p. 2051–2055.

  29. Vicente A, Liu J, Yang GZ. Surface classification based on vibration on omni-wheel mobile base. In: Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2015;916C921.

  30. Murty KSR, Yegnanarayana B. Combining evidence from residual phase and MFCC features for speaker recognition[J]. IEEE Signal Process Lett 2006;13(1):52–55.

    Article  Google Scholar 

  31. Owren MJ, Bernacki RH. Applying linear predictive coding (LPC) to frequency-spectrum analysis of animal acoustic signals[M] Animal acoustic communication. Berlin: Springer; 1998, pp. 129–162.

    Google Scholar 

  32. Muda L, Begam M, Elamvazuthi I. Voice recognition algorithms using mel frequency cepstral coefficient (MFCC) and dynamic time warping (DTW) techniques[J]. arXiv. 2010;1003–4083.

  33. He W, Guan H, Zhang J. Multimodal object recognition from visual and audio sequences. In: 2015 IEEE International Conference on multisensor fusion and integration for intelligent systems (MFI). IEEE;2015. p. 133–138.

  34. Huang GB, Zhu QY, Siew CK. Extreme learning machine: theory and applications [J]. Neurocomputing 2006;70(1):489–501.

    Article  Google Scholar 

  35. Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagation errors. Nature 1986;323:533–536.

    Article  Google Scholar 

  36. Lv Q, Niu X, Dou Y, et al. Classification of hyperspectral remote sensing image using hierarchical local-receptive-field-based extreme learning machine [J]. IEEE Geosci Remote Sens Lett 2016;13(3):434–438.

    Google Scholar 

  37. Li F, Liu H, Xu X. Multi-modal local receptive field extreme learning machine for object recognition[C]. In: 2016 International Joint conference on neural networks (IJCNN). IEEE;2016. pp. 1696–1701.

  38. Xie SJ, Yoon S, Yang J, et al. Feature component-based extreme learning machines for finger vein recognition[J]. Cogn Comput 2014;6(3):446–461.

    Article  Google Scholar 

  39. Chacko BP, Krishnan VRV, Raju G, et al. Handwritten character recognition using wavelet energy and extreme learning machine[J]. Int J Mach Learn Cybern 2012;3(2):149–161.

    Article  Google Scholar 

  40. Fu A, Dong C, Wang L. An experimental study on stability and generalization of extreme learning machines[J]. Int J Mach Learn Cybern 2015;6(1):129–135.

    Article  Google Scholar 

  41. Balasundaram S, Gupta D. On optimization based extreme learning machine in primal for regression and classification by functional iterative method[J]. Int J Mach Learn Cybern 2016;7(5):707–728.

    Article  CAS  Google Scholar 

  42. Sachnev V, Ramasamy S, Sundaram S, et al. A cognitive ensemble of extreme learning machines for steganalysis based on risk-sensitive hinge loss function[J]. Cogn Comput 2015;7(1):103–110.

    Article  Google Scholar 

  43. Landin N, Romano JM, McMahan W, Kuchenbecker KJ. Dimensional reduction of high-frequency accelerations for haptic rendering. Haptics Generating and Perceiving Tangible Sensations. Berlin: Springer; 2010. p. 79–86.

  44. Charles JF. A tutorial on spectral sound processing using Max/MSP and Jitter[J]. Comput Music J 2008;32 (3):87–102.

    Article  Google Scholar 

  45. Serra X, Smith J. Spectral modeling synthesis: a sound analysis/synthesis system based on a deterministic plus stochastic decomposition[J]. Comput Music J 1990;14(4):12–24.

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant U1613212 and Grant 61673238 and in part by the National High-Tech Research and Development Plan under Grant 2015AA042306.

Author information

Authors and Affiliations

  1. State Key Laboratory of Intelligent Technology and Systems, BNRist, Department of Computer Science and Technology, Tsinghua University, Beijing, People’s Republic of China

    Huaping Liu & Fuchun Sun

  2. College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan, China

    Jing Fang & Xinying Xu

Authors
  1. Huaping Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinying Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fuchun Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huaping Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, H., Fang, J., Xu, X. et al. Surface Material Recognition Using Active Multi-modal Extreme Learning Machine. Cogn Comput 10, 937–950 (2018). https://doi.org/10.1007/s12559-018-9571-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-018-9571-z

Keywords

Search

Navigation