Machine Learning

, Volume 71, Issue 1, pp 1–32 | Cite as

Inductive process modeling

  • Will BridewellEmail author
  • Pat Langley
  • Ljupčo Todorovski
  • Sašo Džeroski


In this paper, we pose a novel research problem for machine learning that involves constructing a process model from continuous data. We claim that casting learned knowledge in terms of processes with associated equations is desirable for scientific and engineering domains, where such notations are commonly used. We also argue that existing induction methods are not well suited to this task, although some techniques hold partial solutions. In response, we describe an approach to learning process models from time-series data and illustrate its behavior in three domains. In closing, we describe open issues in process model induction and encourage other researchers to tackle this important problem.


Scientific discovery Process models Compositional modeling System identification Ecosystem modeling 


  1. Arrigo, K. R., Worthen, D. L., & Robinson, D. H. (2003). A coupled ocean-ecosystem model of the Ross Sea: 2. Iron regulation of phytoplankton taxonomic variability and primary production. Journal of Geophysical Research, 108, 3231. CrossRefGoogle Scholar
  2. Asgharbeygi, N., Bay, S., Langley, P., & Arrigo, K. R. (2006). Inductive revision of quantitative process models. Ecological Modelling, 194, 70–79. CrossRefGoogle Scholar
  3. Åström, K. J., & Eykhoff, P. (1971). System identification—a survey. Automatica, 7, 123–167. zbMATHCrossRefGoogle Scholar
  4. Bay, S. D., Shrager, J., Pohorille, A., & Langley, P. (2002). Revising regulatory networks: from expression data to linear causal models. Journal of Biomedical Informatics, 35, 289–297. CrossRefGoogle Scholar
  5. Bechtel, W., & Abrahamsen, A. (2005). Explanation: a mechanistic alternative. Studies in History and Philosophy of the Biological and Biomedical Sciences, 36, 421–441. CrossRefGoogle Scholar
  6. Berryman, A. A. (1992). The origins and evolution of predator–prey theory. Ecology, 73, 1530–1535. CrossRefGoogle Scholar
  7. Box, G., Jenkins, G. M., & Reinsel, G. (1994). Time series analysis: forecasting & control (3rd ed.). Englewood Cliffs: Prentice Hall. Google Scholar
  8. Bradley, E., Easley, M., & Stolle, R. (2001). Reasoning about nonlinear system identification. Artificial Intelligence, 133, 139–188. zbMATHCrossRefGoogle Scholar
  9. Bridewell, W., Sánchez, J. N., Langley, P., & Billman, D. (2006). An interactive environment for the modeling and discovery of scientific knowledge. International Journal of Human–Computer Studies, 64, 1099–1114. CrossRefGoogle Scholar
  10. Bunch, D., Gay, D., & Welsch, R. (1993). Algorithm 717: subroutines for maximum likelihood and quasi-likelihood estimation of parameters in nonlinear regression models. ACM Transactions on Mathematical Software, 19, 109–130. zbMATHCrossRefGoogle Scholar
  11. Cohen, S., & Hindmarsh, A. (1996). CVODE, a stiff/nonstiff ODE solver in C. Computers in Physics, 10, 138–143. Google Scholar
  12. Dennis, J. E. Jr., Gay, D. M., & Welsch, R. E. (1981). An adaptive nonlinear least-squares algorithm. ACM Transactions on Mathematical Software, 7, 348–368. zbMATHCrossRefGoogle Scholar
  13. Dietterich, T. G. (1990). Exploratory research in machine learning. Machine Learning, 5, 5–9. Google Scholar
  14. Domingos, P. (1997). Knowledge acquisition from examples via multiple models. In Proceedings of the fourteenth international conference on machine learning (pp. 98–106). Nashville: Kaufmann. Google Scholar
  15. Džeroski, S., & Todorovski, L. (1993). Discovering dynamics. In Proceedings of the tenth international conference on machine learning (pp. 97–103). Amherst: Kaufmann. Google Scholar
  16. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York City: Chapman & Hall. zbMATHGoogle Scholar
  17. Forbus, K. D. (1984). Qualitative process theory. Artificial Intelligence, 24, 85–168. CrossRefGoogle Scholar
  18. Forbus, K. D., & Falkenhainer, B. (1990). Self-explanatory simulations: an integration of qualitative and quantitative knowledge. In Proceedings of the eighth national conference on artificial intelligence (pp. 380–387). Boston: AAAI Press. Google Scholar
  19. Garrett, S., Coghill, G. M., Srinivasan, A., & King, R. D. (2007). Learning qualitative models of physical and biological systems. In S. D. Džeroski & L. Todorovski (Eds.), Computational discovery of scientific knowledge. Berlin: Springer. Google Scholar
  20. Gay, D. M. (1983). Algorithm 611: Subroutines for unconstrained minimization using a model/trust-region approach. ACM Transactions on Mathematical Software, 9, 503–524. zbMATHCrossRefMathSciNetGoogle Scholar
  21. Ghahramani, Z. (1998). Learning dynamic Bayesian networks. In C. L. Giles & M. Gori (Eds.), Adaptive processing of sequences and data structures. Berlin: Springer. Google Scholar
  22. Ghosh, R., & Tomlin, C. J. (2001). Lateral inhibition through delta-notch signaling: a piecewise affine hybrid model. In Proceedings of the fourth international workshop on hybrid systems: computation and control (pp. 232–246). Springer: Rome. CrossRefGoogle Scholar
  23. Glennan, S. (2002). Rethinking mechanistic explanation. Philosophy of Science, 69, S342–S353. CrossRefGoogle Scholar
  24. Glymour, C., Scheines, R., Spirtes, P., & Kelly, K. (1987). Discovering causal structure: artificial intelligence, philosophy of science, and statistical modeling. San Diego: Academic Press. zbMATHGoogle Scholar
  25. Härdle, W., Horowitz, J., & Kreiss, J. (2003). Bootstrap methods for time series. International Statistical Review, 70, 435–459. Google Scholar
  26. Hempel, C. G., & Oppenheim, P. (1948). Studies in the logic of explanation. Philosophy of Science, 15, 135–175. CrossRefGoogle Scholar
  27. Holling, C. S. (1959). The components of predation as revealed by a study of small-mammal predation of the European pine sawfly. Canadian Entomologist, 91, 293–320. CrossRefGoogle Scholar
  28. Iwasaki, Y., & Simon, H. A. (1994). Causality and model abstraction. Artificial Intelligence, 67, 143–194. zbMATHCrossRefMathSciNetGoogle Scholar
  29. Jost, C., & Ellner, S. (2000). Testing for predator dependence in predator–prey dynamics: a non-parametric approach. Proceedings of the Royal Society of London B, 267, 1611–1620. CrossRefGoogle Scholar
  30. Langley, P. (1981). Data-driven discovery of physical laws. Cognitive Science, 5, 31–54. CrossRefGoogle Scholar
  31. Langley, P., Simon, H. A., Bradshaw, G. L., & Żytkow, J. M. (1987). Scientific discovery: computational explorations of the creative processes. Cambridge: MIT Press. Google Scholar
  32. Lavrač, N. L., & Džeroski, S. D. (1994). Inductive logic programming: techniques and applications. New York City: Ellis Horwood. zbMATHGoogle Scholar
  33. Machamer, P., Darden, L., & Craver, C. F. (2000). Thinking about mechanisms. Philosophy of Science, 67, 1–25. CrossRefMathSciNetGoogle Scholar
  34. Martin, J. H., Gordon, R. M., & Fitzwater, S. E. (1991). The case for iron. Limnology and Oceanography, 36, 1793–1802. Google Scholar
  35. Murray, J. D. (2004). Mathematical biology, I: an introduction (3rd ed.). Berlin: Springer. Google Scholar
  36. Needoba, J. A., & Harrison, P. J. (2004). Influence of low light and a light: dark cycle on NO3 uptake, intracellular NO3 and nitrogen isotope fractionation by marine phytoplankton. Journal of Phycology, 40, 505–516. CrossRefGoogle Scholar
  37. Olson, R. J., Sosik, H. M., Chekalyuk, A. M., & Shalapyonok, A. (2000). Effects of iron enrichment on phytoplankton in the Southern Ocean during late summer: active fluorescence and flow cytometric analyses. Deep-Sea Research Part II-Topical Studies in Oceanography, 47, 3181–3200. CrossRefGoogle Scholar
  38. Ourston, D., & Mooney, R. J. (1990). Changing the rules: a comprehensive approach to theory refinement. In Proceedings of the eighth national conference on artificial intelligence (pp. 815–820). Boston: AAAI Press. Google Scholar
  39. Pazzani, M. J., Mani, S., & Shankle, W. R. (2001). Acceptance by medical experts of rules generated by machine learning. Methods of Information in Medicine, 40, 380–385. Google Scholar
  40. Poritz, A. (1988). Hidden Markov models: a guided tour. In Proceedings of the international conference on acoustic, speech and signal processing (pp. 7–13). New York City: IEEE Press. Google Scholar
  41. Schwabacher, M., & Langley, P. (2001). Discovering communicable scientific knowledge from spatio-temporal data. In Proceedings of the eighteenth international conference on machine learning (pp. 489–496). Williamstown: Kaufmann. Google Scholar
  42. Simon, H. A. (1954). Spurious correlation: a causal interpretation. Journal of the American Statistical Association, 49, 467–479. zbMATHCrossRefGoogle Scholar
  43. Todorovski, L. (2003). Using domain knowledge for automated modeling of dynamic systems with equation discovery. Doctoral dissertation, Faculty of Computer and Information Science, University of Ljubljana. Ljubljana, Slovenia. Google Scholar
  44. Todorovski, L., & Džeroski, S. (1997). Declarative bias in equation discovery. In Proceedings of the fourteenth international conference on machine learning (pp. 376–384). Nashville: Kaufmann. Google Scholar
  45. Veilleux, B. G. (1979). An analysis of predatory interaction between paramecium and didinium. Journal of Animal Ecology, 48, 787–803. CrossRefGoogle Scholar
  46. Washio, T., Motoda, H., & Niwa, Y. (2000). Enhancing the plausibility of law equation discovery. In Proceedings of the seventeenth international conference on machine learning (pp. 1127–1134). Stanford: Kaufmann. Google Scholar
  47. Williams, R., & Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1, 270–280. CrossRefGoogle Scholar
  48. Woodward, J. (2002). What is a mechanism? A counterfactual account. Philosophy of Science, 69, S366–S377. CrossRefGoogle Scholar
  49. Zheng, J., Vankataramanan, L., & Sigworth, F. J. (2001). Hidden Markov model analysis of intermediate gating steps associated with the pore gate of Shaker potassium channels. Journal of General Physiology, 118, 547–562. CrossRefGoogle Scholar
  50. Żytkow, J. M., Zhu, J., & Hussam, A. (1990). Automated discovery in a chemistry laboratory. In Proceedings of the eighth national conference on artificial intelligence (pp. 889–894). Boston: AAAI Press. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • Will Bridewell
    • 1
    Email author
  • Pat Langley
    • 1
  • Ljupčo Todorovski
    • 2
  • Sašo Džeroski
    • 2
  1. 1.Computational Learning Laboratory, Center for the Study of Language and InformationStanford UniversityStanfordUSA
  2. 2.Department of Knowledge TechnologiesJozef Stefan InstituteLjubljanaSlovenia

Personalised recommendations