A Learning Approach for Ill-Posed Optimisation Problems

  • Jörg FrochteEmail author
  • Stephen Marsland
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1127)


Supervised learning can be thought of as finding a mapping between spaces of input and output vectors. In the case that the function to be learned is multi-valued (so that there are several correct output values for a given input) the problem becomes ill-posed, and many standard methods fail to find good solutions. However, optimisation problems based on multi-valued functions are relatively common. They include reverse robot kinematics, and the research field of AutoML – which is becoming increasingly popular – where one seeks to establish optimal hyperparameters for a learning algorithm for a particular problem based on loss function values for trained networks, or to reuse training from previous networks. We present an analysis of this problem, together with an approach based on k-nearest neighbours, which we demonstrate on a set of simple examples, including two application areas of interest.


Multi-valued functions Ill-posed optimisation Local models AutoML 


  1. 1.
    Brouwer, R.K.: Feed-forward neural network for one-to-many mappings using fuzzy sets. Neurocomputing 57, 345–360 (2004)CrossRefGoogle Scholar
  2. 2.
    Butterfield, J., Osentoski, S., Jay, G., Jenkins, O.C.: Learning from demonstration using a multi-valued function regressor for time-series data. In: 2010 10th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 328–333. IEEE (2010)Google Scholar
  3. 3.
    Damas, B., Santos-Victor, J.: Online learning of single-and multivalued functions with an infinite mixture of linear experts. Neural Comput. 25(11), 3044–3091 (2013)CrossRefGoogle Scholar
  4. 4.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pp. 226–231 (1996)Google Scholar
  5. 5.
    Goldman, S.A., Rivest, R.L., Schapire, R.E.: Learning binary relations and total orders. SIAM J. Comput. 22(5), 1006–1034 (1993)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Hahn, K., Waschulzik, T.: On the use of local RBF networks to approximate multivalued functions and relations. In: Niklasson, L., Bodén, M., Ziemke, T. (eds.) ICANN 1998. PNC, pp. 505–510. Springer, London (1998). Scholar
  7. 7.
    Hecht-Nielsen, R.: Theory of the backpropagation neural network. In: Neural Networks for Perception, pp. 65–93. Elsevier (1992)Google Scholar
  8. 8.
    Kilp, M., Knauer, U., Mikhalev, A.: Monoids, Acts and Categories, With Applications to Wreath Products and Graphs. de Gruyter (2000)Google Scholar
  9. 9.
    Lee, K.W., Lee, T.: Design of neural networks for multi-value regression. In: 2001 Proceedings of International Joint Conference on Neural Networks, IJCNN 2001, vol. 1, pp. 93–98. IEEE (2001)Google Scholar
  10. 10.
    Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Shizawa, M.: Multivalued regularization network-a theory of multilayer networks for learning many-to-h mappings. Electron. Commun. Jpn. (Part III: Fundam. Electron. Sci.) 79(9), 98–113 (1996)CrossRefGoogle Scholar
  12. 12.
    Sussmann, H.J.: Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Netw. 5(4), 589–593 (1992)CrossRefGoogle Scholar
  13. 13.
    Tomikawa, Y., Nakayama, K.: Approximating many valued mappings using a recurrent neural network. In: Proceedings of the 1998 IEEE International Joint Conference on Neural Networks, IEEE World Congress on Computational Intelligence, vol. 2, pp. 1494–1497. IEEE (1998)Google Scholar
  14. 14.
    Wong, C., Houlsby, N., Lu, Y., Gesmundo, A.: Transfer learning with neural AutoML. In: Advances in Neural Information Processing Systems, pp. 8356–8365 (2018)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Bochum University of Applied SciencesHeiligenhausGermany
  2. 2.Victoria University of WellingtonWellingtonNew Zealand

Personalised recommendations