Abstract
This chapter introduces several meta-heuristics and learning methods, which will be employed in later chapters. These methods will be employed to extend evolutionary computation frameworks in later chapters. Readers familiar with these methods may skip this chapter.
Keywords
- Particle swarm optimization (PSO)
- Differential evolution (DE)
- k-means algorithm
- Support vector machine (SVM)
- Relevance vector machine (RVM)
- k-nearest neighbor classifier (k-NN)
- Transfer learning
- Bagging
- Boosting
- Gröbner bases
- Affinity propagation
- Convolutional neural networks (CNN)
- Generative adversary networks (GAN)
- Bayesian networks
- Loopy belief propagation
I remember the first time I met Edsger Dijkstra. ... He asked me what I was working on. Perhaps just to provoke a memorable exchange I said, “AI.” To that he immediately responded, “Why don’t you work on I?”
(Leslie Valiant, Probably Approximately Correct: Nature’s Algorithms for Learning and Prospering in a Complex, Basic Books 2014)
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Given a set of vectors \(\mathbf {v}_1, \ldots , \mathbf {v}_n\), its Gram matrix G is an \(n\times n\) matrix, whose element is an inner product of two vectors, i.e., \(G_{ij} = \mathbf {v}_i \cdot \mathbf {v}_j\).
- 2.
The point P is in the range which is outside a circle (the center is a mid-point of centers of circle \(O_{1}\) and circle \(O_{2}\), and the radius is \(\frac{\mid r_{1} - r_{2}\mid }{2}\)), and which is inside a circle (the center is the same, and the radius is\(\frac{(r_{1} +r_{2})}{2}\)).
- 3.
A degree measure of the similarity between two documents A and B, i.e., \(\frac{\mid A \cap B\mid }{\mid A \cup B \mid }\).
- 4.
The used parameters are as follows: 300 sample points, cluster centers \(=\) [[2, 2], [−2, −2], [2, −2]], and cluster std. \(=\) 0.5.
- 5.
The first, second, and fifth of these layers are linked to max pooling.
- 6.
The softmax activation function is defined as \(f_i(x_1,\ldots ,x_d) = \frac{\exp (x_i)}{\sum _j \exp (x_j)}\;(i=1,\ldots ,d)\), where d is the unit number of the layer.
- 7.
Principal component analysis of the RGB values in the data set is performed and then random changes are made, centered on these axes, thereby preserving the essential features of the original natural images.
- 8.
ILSVRC2012 dataset is a subset of the ImageNet database [40]. It contains 1,000 object categories. The training set, validation set, and testing set contain 1.3 M, 50 K, and 150 K images, respectively. The input images are of \(224\times 224\times 3\) pixels.
References
Aha, D.W., Kibler, D., Albert, M.: Instance-based learning algorithms. Mach. Learn. 6, 37–66 (1991)
Angeline, P.J.: Evolutionary optimization versus particle swarm optimization: philosophy and performance differences. In: Porto, V.W., Saravanan, N., Waagen, D., Eiben, A.E. (eds.) Evolutionary Programming VII, pp. 601–610. Springer, Berlin (1998)
Bäck, T., Fogel, D.B., Michalewicz, Z. (eds.): Evolutionary Computation 1: Basic Algorithms and Operators. Institute of Physics Publishing, Bristol (2000)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)
Buchberger, B.: A criterion for detecting unnecessary reductions in the construction of Gröbner-bases. In: Proceedings of the International Symposium on Symbolic and Algebraic Computation (EUROSAM’79), pp. 3–21 (1979)
Buchberger, B.: A note on the complexity of constructing Gröbner-bases. In: van Hulzen, J.A. (ed.) Computer Algebra, EUROCAL 1983. Lecture Notes in Computer Science, vol. 162. Springer, Berlin (1983)
Buchberger, B.: A critical pair completion algorithm for finitely generated ideals in rings. In: Borger, E., Hasenjaeger, G., Rodding, D. (eds.) Logic and Machines: Decision Problems and Complexity. Lecture Notes in Computer Science, vol. 171. Springer, Berlin (1984)
Buchberger, B.: Gröbner bases: an algorithmic method in polynomial ideal theory. In: Bose, N.K. (ed.) Multidimensional Systems Theory. D. Reidel Publishing Company, Dordrecht (1985)
Buchberger, B.: Applications of Gröbner bases in non-linear computational geometry. In: Jansen, R. (ed.) Trends in Computer Algebra. Lecture Notes in Computer Science, vol. 296. Springer, Berlin (1987)
Buchberger, B., Winkler, F.: Gröbner Bases and Applications, vol. 251. Cambridge University Press, Cambridge (1998)
Caviness, B.F., Johnson, J.R. (eds.): Quantifier Elimination and Cylindrical Algebraic Decomposition. Springer, Berlin (2013)
Chakraborti, N., Misra, K., Bhatt, P., Barman, N., Prasad, R.: Tight-binding calculations of Si-H clusters using genetic algorithms and related techniques: studies using differential evolution. J. Phase Equilibria 22(5), 525–530 (2001)
Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)
Chou, S., Schelter, W.F.: Proving geometry theorems with rewrite rules. J. Autom. Reason. 2, 253–273 (1986)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)
Dasarathy, B.: Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques. IEEE Computer Society Press, Los Alamitos (1991)
Drucker, H.: Improving Regression Using Boosting Techniques. In: Proceedings of International Conference on Machine Learning (ICML97) (1997)
Dueck, D., Frey, B.J.: Non-metric affinity propagation for unsupervised image categorization. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8. IEEE (2007)
Eberhart, R.C., Shi, Y.: Comparison between genetic algorithms and particle swarm optimization. In: Proceedings of the Seventh Annual Conference on Evolutionary Programming, pp. 611–619 (1998)
Fan, R.-E., Chang, L.-W., Hsieh, C.-J., Wang, X.-R., Lin, C.-J.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)
Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70, 41–54 (2006)
Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Proceedings of International Conference on Machine Learning (ICML96) (1996)
Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315, 972–976 (2007)
Fukushima, K.: Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36(4), 193–202 (1980)
Gämperle, R., Müller, S.D., Koumoutsakos, P.: A parameter study for differential evolution. In: Proceedings of International Conference on Advances in Intelligent Systems, Fuzzy Systems, Evolutionary Computation, pp. 293–298 (2002)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Higashi, N., Iba, H.: Particle swarm optimization with Gaussian mutation. In: Proceedings of IEEE Swarm Intelligence Symposium (SIS03), pp. 72–79 (2003)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)
Holland, J.H.: Adaptation in Natural and Artificial Systems. University of Michigan Press (1975)
Hu, R., Namee, B.M., Delany, S.J.: Off to a good start: using clustering to select the initial training set in active learning. In: Proceedings of the Florida Artificial Intelligence Research Society Conference (FLAIRS) (2010)
Huang, C.-H., Wang, C.-J.: A GA-based feature selection and parameters optimization for support vector machines. Expert Syst. Appl. 31, 231–240 (2006)
Iba, H., Hirochika Inoue, H.: Reasoning of geometric concepts based on algebraic constraint-directed method. In: Proceedings of the IJCAI, pp. 143–151 (1991)
Iba, H., Noman, N.: New Frontiers in Evolutionary Algorithms: Theory and Applications. World Scientific Publishing Company, Singapore (2011)
Jia, Y., Wang, J., Zhang, C., Hua, X.-S.: Finding image exemplars using fast sparse affinity propagation. In: Proceedings of the 16th ACM International Conference on Multimedia, pp. 639–642. ACM (2008)
Kapur, D.: Geometric reasoning and artificial intelligence: introduction to the special volume. Artif. Intell. 37 (1988)
Kennedy, J., Eberhart, R.C.: Particle swarm optimization. In: Proceedings of IEEE the International Conference on Neural Networks, pp. 1942–1948 (1995)
Kennedy, J., Eberhart, R.C.: Swarm Intelligence. Morgan Kaufmann Publishers, San Francisco (2001)
Kennedy, J., Spears, W.M.: Matching algorithms to problems: an experimental test of the particle swarm and some genetic algorithms on the multimodal problem generator. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pp. 78–83 (1998)
Krink, T., Filipič, B., Fogel, G., Thomsen, R.: Noisy optimization problems – a particular challenge for differential evolution? In: Proceedings of Congress on Evolutionary Computation, pp. 332–339 (2004)
Krizhevsky, A., Sutskerver, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25 (NIPS), pp. 1097–1105 (2012)
Larsen, A.B.L., S\(\phi \)nderby, S.K., Winther, O.: Autoencoding beyond pixels using a learned similarity metric (2015). arXiv:1512.09300
Laubenbacher, R., Stigler, B.: A computational algebra approach to the reverse engineering of gene regulatory networks. J. Theor. Biol. 229, 523–537 (2004)
Le, Q., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G., Dean, J., Ng, A.: Building high-level features using large scale unsupervised learning. In: Proceedings of the 29th International Conference on Machine Learning (2012)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Limbeck, J.: Computation of approximate border bases and applications, Ph.D. thesis, Passau, Universität Passau, Dissertation (2014)
Liu, Z., Li, P., Zheng, Y., Sun, M.: Clustering to find exemplar terms for key phrase extraction. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, vol. 1, pp. 257–266. Association for Computational Linguistics (2009)
Loos, R.: Introduction. In: Buchberger, B., et al. (eds.) Computer Algebra Symbolic and Algebraic Computation. Springer, Berlin (1982)
Maclin, R., Opitz, D.: An empirical evaluation of bagging and boosting. In: Proceedings of National Conference on Artificial Intelligence (AAAI97) (1997)
Möller, H.M., Buchberger, B.: The construction of multivariate polynomials with preassigned zeros. In: Proceedings of the European Computer Algebra Conference on Computer Algebra, pp. 24–31 (1982)
Mourrain, B.: A new criterion for normal form algorithms. Applied Algebra, Algebraic Algorithms and Error-correcting Codes, pp. 430–442. Springer, Berlin (1999)
Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427–436 (2015)
Noman, N., Iba, H.: Enhancing differential evolution performance with local search for high dimensional function optimization. In: Proceedings of Genetic and Evolutionary Computation Conference (GECCO2005), pp. 967–974 (2005)
Noman, N., Iba, H.: A new generation alternation model for differential evolution. In: Proceedings of Genetic and Evolutionary Computation Conference (GECCO 2006), pp. 1265–1272 (2006)
Noman, N., Iba, H.: Differential evolution for economic load dispatch problems. Elsevier Electric Power Syst. Res. 78(8), 1322–1331 (2008)
Noman, N., Iba, H.: Accelerating differential evolution using an adaptive local search. IEEE Trans. Evol. Comput. 12(1), 107–125 (2008)
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Paul, T.K., Ueno, K., Iwata, K., Hayashi, T., Honda, N.: Genetic algorithm based methods for identification of health risk factors aimed at preventing metabolic syndrome. In: Proceedings of the 7th International Conference on Simulated Evolution And Learning (SEAL’08). LNCS, vol. 5361, pp. 210–219. Springer, Berlin (2008)
Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo (1988)
Price, K.V., Storn, R.M., Lampinen, J.A.: Differential Evolution: A Practical Approach to Global Optimization. Springer, Berlin (2005)
Quinlan, J.R.: Bagging, Boosting, and C4.5. In: Proceedings of National Conference on Artificial Intelligence (AAAI96) (1996)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks (2015). arXiv:1511.06434
Sato, H., Hasegawa, Y., Bollegala, D., Iba, H.: Improved sampling using loopy belief propagation for probabilistic model building genetic programming. Swarm Evol. Comput. 23, 1–10 (2015)
Stifter, S.: Algebraic methods for computing inverse kinematics. J. Intell. Robot. Syst. 11(1–2), 79–89 (1994)
Storn, R.: System design by constraint adaptation and differential evolution. IEEE Trans. Evol. Comput. 3(1), 22–34 (1999)
Storn, R., Price, K.V.: Differential evolution – a simple and efficient adaptive scheme for global optimization over continuous spaces. Technical report TR-95-012, ICSI (1995)
Storn, R., Price, K.V.: Differential evolution -a simple and efficient heuristic for global optimization over continuous spaces. J. Global Optim. 11(4), 341–359 (1997)
Swan, J., Neumann, G.K., Krawiec, K.: Analysis of semantic building blocks via Grobner bases. In: Johnson, C., Krawiec, K., O’Neill, M., Moraglio, A. (eds.) Semantic Methods in Genetic Programming (SMGP) at Parallel Problem Solving from Nature (PPSN XIV), Ljubljana, Slovenia (2014)
Tipping, M.E.: The relevance vector machine. Advances in Neural Information Processing Systems, pp. 652–658. MIT Press, Cambridge (2000)
Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)
Yang, Y., Iba, H.: Fooling voice based on evolutionary computation. In: Proceedings of Evolutionary Computation Symposium, Dec. 9–10, Hokkaido, Japan (2017)
Zaharie, D.: Critical values for the control parameters of differential evolution algorithms. In: Proceedings of MENDEL 2002, 8th International Conference on Soft Computing, pp. 62–67 (2002)
Zha, Z.-J., Yang, L., Mei, T., Wang, M., Wang, Z.: Visual query suggestion. In: Proceedings of the 17th ACM International Conference on Multimedia, pp. 15–24. ACM (2009)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Iba, H. (2018). Meta-heuristics, Machine Learning, and Deep Learning Methods. In: Evolutionary Approach to Machine Learning and Deep Neural Networks. Springer, Singapore. https://doi.org/10.1007/978-981-13-0200-8_2
Download citation
DOI: https://doi.org/10.1007/978-981-13-0200-8_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-0199-5
Online ISBN: 978-981-13-0200-8
eBook Packages: Computer ScienceComputer Science (R0)