Skip to main content
SpringerLink
Log in

Optimizing Echo State Networks for Static Pattern Recognition

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Static pattern recognition requires a machine to classify an object on the basis of a combination of attributes and is typically performed using machine learning techniques such as support vector machines and multilayer perceptrons. Unusually, in this study, we applied a successful time-series processing neural network architecture, the echo state network (ESN), to a static pattern recognition task. The networks were presented with clamped input data patterns, but in this work, they were allowed to run until their output units delivered a stable set of output activations, in a similar fashion to previous work that focused on the behaviour of ESN reservoir units. Our aim was to see if the short-term memory developed by the reservoir and the clamped inputs could deliver improved overall classification accuracy. The study utilized a challenging, high dimensional, real-world plant species spectroradiometry classification dataset with the objective of accurately detecting one of the world’s top 100 invasive plant species. Surprisingly, the ESNs performed equally well with both unsettled and settled reservoirs. Delivering a classification accuracy of 96.60%, the clamped ESNs outperformed three widely used machine learning techniques, namely support vector machines, extreme learning machines and multilayer perceptrons. Contrary to past work, where inputs were clamped until reservoir stabilization, it was found that it was possible to obtain similar classification accuracy (96.49%) by clamping the input patterns for just two repeats. The chief contribution of this work is that a recurrent architecture can get good classification accuracy, even while the reservoir is still in an unstable state.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Alexandre LA, Embrechts MJ. Reservoir size, spectral radius and connectivity in static classification problems. In: Alippi C, Polycarpou M, Panayiotou C, Ellinas G, editors. Artificial neural networks-ICANN 2009, vol. 5768. Berlin Heidelberg: Springer; 2009. p. 1015–24.

    Chapter  Google Scholar 

  2. L. A. Alexandre, M. J. Embrechts, and J. Linton, “Benchmarking reservoir computing on time-independent classification tasks,” in International Joint Conference on Neural Networks, 2009. IJCNN 2009., pp. 89–93.

  3. Bache K, Lichman M. UCI machine learning repository. Irvine, School of Information and Computer Sciences: University of California; 2013.

    Google Scholar 

  4. Bacic B. Echo state network for 3D motion pattern indexing: a case study on tennis forehands. In: Braunl T, McCane B, Rivera M, Yu X, editors. Image and video technology: 7th Pacific-rim Symposium, PSIVT 2015, Auckland, New Zealand, November 25–27, 2015, revised selected papers. Cham: Springer International Publishing; 2016. p. 295–306.

    Chapter  Google Scholar 

  5. Bozhkov L, Koprinkova-Hristova P, Georgieva P. Reservoir computing for emotion valence discrimination from EEG signals. Neurocomputing. 2017;231:28–40.

    Article  Google Scholar 

  6. Busing L, Schrauwen B, Legenstein R. Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural Comput. May 2010;22(5):1272–311.

    Article  PubMed  Google Scholar 

  7. Butcher JB, Day CR, Haycock PW, Verstraeten D, Schrauwen B. Defect detection in reinforced concrete using random neural architectures. Computer-Aided Civil and Infrastructure Engineering. 2014;29(3):191–207.

    Article  Google Scholar 

  8. Chang C-C, Lin C-J. LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol. 2011;2(3):27:1–27:27.

    Article  Google Scholar 

  9. Cho MA, Sobhan I, Skidmore AK, de Leeuw J. Discriminating species using hyperspectral indices at leaf and canopy scales. In: ISPRS 2008 : Proceedings of the XXI congress: silk road for information from imagery: the International Society for Photogrammetry and Remote Sensing; 2008. p. 369–76.

    Google Scholar 

  10. Dutoit X, Schrauwen B, Campenhout JV, Stroobandt D, Brussel HV, Nuttin M. Pruning and regularization in reservoir computing. Neurocomputing. 2009;72(7–9):1534–46.

    Article  Google Scholar 

  11. Edwards C, Taylor SL. A survey and strategic appraisal of rhododendron invasion and control in woodwood areas in Argyll and Bute. Forestry Commision Scotland: Perth Conservancy; 2008.

    Google Scholar 

  12. M. J. Embrechts, L. A. Alexandre, and J. D. Linton, “Reservoir computing for static pattern recognition,” in European Symposium on Artificial Neural Networks (ESANN) 2009, 2009.

  13. Emmerich C, Reinhart R, Steil J. Recurrence enhances the spatial encoding of static inputs in reservoir networks. In: Diamantaras K, Duch W, Iliadis L, editors. Artificial neural networks-ICANN, vol. 6353. Berlin Heidelberg, 2010: Springer; 2010. p. 148–53.

    Google Scholar 

  14. Gaulton R, Olaya G, Wallington ED, Malthus TJ. Continuous cover forestry sensing in the UK? Quantifying Forest structure using remote sensing. In: Proceedings of ForestSAT Conference; 2005.

    Google Scholar 

  15. He KS, Rocchini D, Neteler M, Nagendra H. Benefits of hyperspectral remote sensing for tracking plant invasions. Divers Distrib. 2011;17(3):381–92.

    Article  Google Scholar 

  16. Hoerl AE, Kennard RW. Ridge regression: biased estimation for Nonorthogonal problems. Technometrics. 1970;12(1):55–67.

    Article  Google Scholar 

  17. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme learning machine: a new learning scheme of feedforward neural networks,” in Neural Networks, 2004. Proceedings. 2004 I.E. International Joint Conference on, 2004, vol. 2, pp. 985–990 vol. 2.

  18. Jaeger H. The ‘echo state’ approach to analysing and training recurrent neural networks: Fraunhofer Institute for Autonomous Intelligent Systems; 2001.

  19. H. Jaeger, “A tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the ‘echo state network’ approach,” Fraunhofer Institute for Autonomous Intelligent Systems (AIS), 4, 2013.

  20. Lukosevicius M. A practical guide to applying echo state networks. In: Montavon G, Orr GB, Muller K-R, editors. Neural networks: tricks of the trade, vol. 7700. 2nd ed. Berlin Heidelberg: Eds. Springer; 2012. p. 659–86.

    Chapter  Google Scholar 

  21. Matthews BW. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure. 1975;405(2):442–51.

    Article  CAS  Google Scholar 

  22. Meftah B, Lezoray O, Benyettou A. Novel approach using Echo state networks for microscopic cellular image segmentation. Cogn Comput. 2016;8(2):237–45.

    Article  Google Scholar 

  23. Montgomery DC, Peck EA, Vining CG. Introduction to linear regression analysis: Wiley; 1982.

  24. Natschlager T, Maass W, Markram H. The ‘liquid Computer’: a novel strategy for real-time computing on time series. Special Issue on Foundations of Information Processing of TELEMATIK. 2002;8(1):39–43.

    Google Scholar 

  25. R. F. Reinhart and J. J. Steil, “Attractor-based computation with reservoirs for online learning of inverse kinematics,” in European Symposium on Artificial Neural Networks (ESANN) 2009, 2009.

  26. Scardapane S, Uncini A. Semi-supervised Echo state networks for audio classification. Cogn Comput. 2016:1–11.

  27. Sillin HO, Aguilera R, Shieh H-H, Avizienis AV, Aono M, Stieg AZ, Gimzewski JK. A theoretical and experimental study of neuromorphic atomic switch networks for reservoir computing. Nanotechnology. 2013;24(38):384004.

    Article  PubMed  Google Scholar 

  28. Taylor SL, Hill RA, Edwards C. Characterising invasive non-native Rhododendron Ponticum spectra signatures with spectroradiometry in the laboratory and field: potential for remote mapping. ISPRS J Photogramm Remote Sens. 2013;81(0):70–81.

    Article  Google Scholar 

  29. Tong MH, Bickett AD, Christiansen EM, Cottrell GW. Learning grammatical structure with Echo state networks. Neural Netw. 2007;20(3):424–32.

    Article  PubMed  Google Scholar 

  30. Verstraeten D, Schrauwen B, D’Haene M, Stroobandt D. An experimental unification of reservoir computing methods. Neural Netw. 2007;20(3):391–403.

    Article  CAS  PubMed  Google Scholar 

  31. Wang L. Invasive species spread mapping using multi-resolution remote sensing data. Int Arch Photogramm Remote Sens Spat Inf Sci. 2008;37:135–42.

    Google Scholar 

  32. Williams F, Eschen R, Harris A, Djeddour D, Pratt C, Shaw RS, Varia S, Lamontagne-Godwin J, Thomas SE, Murphy ST. The economic cost of invasive non-native species on great Britain: CABI; 2010.

  33. A. J. Wootton, C. R. Day, and P. W. Haycock, “Echo State Network Applications in Structrual Health Monitoring,” in Proceedings of the 53rd Annual Conference of The British Institute of Non-Destructive Testing (NDT 2014), 2014, pp. 289–300.

Download references

Acknowledgements

The authors thank NERC FSF for the provision of an equipment grant to S. L. Taylor, which was used to capture the original leaf sample spectroradiometry dataset.

Author information

Authors and Affiliations

  1. Foundation Year Centre, Keele University, Keele, Staffordshire, ST5 5BG, UK

    Adam J. Wootton & Peter W. Haycock

  2. School of Computing and Mathematics, Keele University, Keele, Staffordshire, ST5 5BG, UK

    Adam J. Wootton & Charles R. Day

  3. School of Life Sciences, Keele University, Keele, Staffordshire, ST5 5BG, UK

    Sarah L. Taylor

Authors
  1. Adam J. Wootton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah L. Taylor
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charles R. Day
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peter W. Haycock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adam J. Wootton.

Ethics declarations

Conflict of Interest

Adam Wootton declares that he has no conflict of interest. Sarah Taylor received a research grant from NERC FSF. Charles Day declares that he has no conflict of interest. Peter Haycock declares that he has no conflict of interest.

Funding

S. L. Taylor has received a research grant from NERC FSF, which was used to capture the original leaf sample spectroradiometry dataset.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wootton, A.J., Taylor, S.L., Day, C.R. et al. Optimizing Echo State Networks for Static Pattern Recognition. Cogn Comput 9, 391–399 (2017). https://doi.org/10.1007/s12559-017-9468-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-017-9468-2

Keywords

Search

Navigation