Semi-supervised Regression and System Identification,

  • Henrik Ohlsson
  • Lennart Ljung


System Identification and Machine Learning are developing mostly as independent subjects, although the underlying problem is the same: To be able to associate “outputs” with “inputs”. Particular areas in machine learning of substantial current interest are manifold learning and unsupervised and semi-supervised regression. We outline a general approach to semi-supervised regression, describe its links to Local Linear Embedding, and illustrate its use for various problems. In particular, we discuss how these techniques have a potential interest for the system identification world.


Blood Oxygenation Level Dependent Regression Problem Unlabeled Data Lower Dimensional Manifold fMRI Measurement 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bauwens, M., Ohlsson, H., Barbé, K., Beelaerts, V., Dehairs, F., Schoukens, J.: On climate reconstruction using bivalve shells: Three methods to interpret the chemical signature of a shell. In: 7th IFAC Symposium on Modelling and Control in Biomedical Systems (April 2009)Google Scholar
  2. 2.
    Belkin, M., Niyogi, P., Sindhwani, V.: Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. J. Mach. Learn. Res. 7, 2399–2434 (2006)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Bengio, Y., Delalleau, O., Le Roux, N.: Label propagation and quadratic criterion. In: Chapelle, O., Schölkopf, B., Zien, A. (eds.) Semi-Supervised Learning, pp. 193–216. MIT Press, Cambridge (2006)Google Scholar
  4. 4.
    Chapelle, O., Schölkopf, B., Zien, A. (eds.): Semi-Supervised Learning. MIT Press, Cambridge (2006)Google Scholar
  5. 5.
    Cybenko, G.: Just-in-time learning and estimation. In: Bittanti, S., Picci, G. (eds.) Identification, Adaptation, Learning. The Science of Learning Models from data. NATO ASI Series, pp. 423–434. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  6. 6.
    de Ridder, D., Duin, R.: Locally linear embedding for classification. Tech Report, PH-2002-01, Pattern Recognition Group, Dept. of Imaging Science & Technology, Delft University of Technology, Delft, The Netherlands (2002)Google Scholar
  7. 7.
    de Ridder, D., Kouropteva, O., Okun, O., Pietikäinen, M.: Supervised locally linear embedding. In: Kaynak, O., Alpaydın, E., Oja, E., Xu, L. (eds.) ICANN 2003 and ICONIP 2003. LNCS, vol. 2714, pp. 333–341. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  8. 8.
    Gillikin, D.P., Dehairs, F., Lorrain, A., Steenmans, D., Baeyens, W., André, L.: Barium uptake into the shells of the common mussel (Mytilus edulis) and the potential for estuarine paleo-chemistry reconstruction. Geochimica et Cosmochimica Acta 70(2), 395–407 (2006)CrossRefGoogle Scholar
  9. 9.
    Gillikin, D.P., Lorrain, A., Bouillon, S., Willenz, P., Dehairs, F.: Stable carbon isotopic composition of Mytilus edulis shells: relation to metabolism, salinity, \(\delta^{13}\mbox{C}_{DIC}\) and phytoplankton. Organic Geochemistry 37(10), 1371–1382 (2006)CrossRefGoogle Scholar
  10. 10.
    Goldberg, A.B., Zhu, X.: Seeing stars when there aren’t many stars: Graph-based semi-supervised learning for sentiment categorization. In: HLT-NAACL 2006 Workshop on Textgraphs: Graph-based Algorithms for Natural Language Processing (2006)Google Scholar
  11. 11.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer, New York (2001)zbMATHGoogle Scholar
  12. 12.
    Hosmer Jr., D.W.: A comparison of iterative maximum likelihood estimates of the parameters of a mixture of two normal distributions under three different types of sample. Biometrics 29(4), 761–770 (1973)CrossRefGoogle Scholar
  13. 13.
    Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences, vol. 30. Springer, Berlin (1995)CrossRefGoogle Scholar
  14. 14.
    Merz, C.J., St. Clair, D.C., Bond, W.E.: Semi-supervised adaptive resonance theory (smart2). In: International Joint Conference on Neural Networks, IJCNN, Jun. 1992, vol. 3, pp. 851–856 (1992)Google Scholar
  15. 15.
    Narendra, K.S., Li, S.-M.: Neural networks in control systems. In: Smolensky, P., Mozer, M.C., Rumelhard, D.E. (eds.) Mathematical Perspectives on Neural Networks, pp. 347–394. Lawrence Erlbaum Associates, Mahwah (1996)Google Scholar
  16. 16.
    Navaratnam, R., Fitzgibbon, A.W., Cipolla, R.: The joint manifold model for semi-supervised multi-valued regression. In: IEEE 11th International Conference on Computer Vision, ICCV 2007, October 2007, pp. 1–8 (2007)Google Scholar
  17. 17.
    Nigam, K., McCallum, A., Thrun, S., Mitchell, T.: Learning to classify text from labeled and unlabeled documents. In: AAAI ’98/IAAI ’98: Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, pp. 792–799. AAAI Press, Menlo Park (1998)Google Scholar
  18. 18.
    Ohlsson, H., Roll, J., Ljung, L.: Manifold-constrained regressors in system identification. In: Proc. 47st IEEE Conference on Decision and Control, December 2008, pp. 1364–1369 (2008)Google Scholar
  19. 19.
    Putten, E.V., Dehairs, F., André, L., Baeyens, W.: Quantitative in situ microanalysis of minor and trace elements in biogenic calcite using infrared laser ablation - inductively coupled plasma mass spectrometry: a critical evaluation. Analytica Chimica Acta 378(1-3), 261–272 (1999)CrossRefGoogle Scholar
  20. 20.
    Rahimi, A., Recht, B., Darrell, T.: Learning to transform time series with a few examples. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(10), 1759–1775 (2007)CrossRefGoogle Scholar
  21. 21.
    Roweis, S.T., Saul, L.K.: Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science 290(5500), 2323–2326 (2000)CrossRefGoogle Scholar
  22. 22.
    olde Scheper, T., Klinkenberg, D., Pennartz, C., van Pelt, J.: A Mathematical Model for the Intracellular Circadian Rhythm Generator. J. Neurosci. 19(1), 40–47 (1999)Google Scholar
  23. 23.
    Stenman, A.: Model on Demand: Algorithms, Analysis and Applications. Linköping Studies in science and Technology. Thesis No 571, Linköping University, SE-581 83 Linköping, Sweden (April 1999)Google Scholar
  24. 24.
    Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)CrossRefGoogle Scholar
  25. 25.
    Wang, F., Zhang, C.: Label propagation through linear neighborhoods. IEEE Transactions on Knowledge and Data Engineering 20(1), 55–67 (2008)CrossRefGoogle Scholar
  26. 26.
    Yang, X., Fu, H., Zha, H., Barlow, J.: Semi-supervised nonlinear dimensionality reduction. In: ICML ’06: Proceedings of the 23rd international conference on Machine learning, pp. 1065–1072. ACM, New York (2006)CrossRefGoogle Scholar
  27. 27.
    Zhao, L., Zhang, Z.: Supervised locally linear embedding with probability-based distance for classification. Comput. Math. Appl. 57(6), 919–926 (2009)zbMATHCrossRefGoogle Scholar
  28. 28.
    Zhu, X.: Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison (2005)Google Scholar

Copyright information

© Springer Berlin Heidelberg 2010

Authors and Affiliations

  • Henrik Ohlsson
    • 1
  • Lennart Ljung
    • 1
  1. 1.Division of Automatic Control, Department of Electrical EngineeringLinköpings UniversitetLinköpingSweden

Personalised recommendations