Skip to main content

Hybrid Classification and Reasoning for Image-Based Constraint Solving

  • Conference paper
  • First Online:
Integration of Constraint Programming, Artificial Intelligence, and Operations Research (CPAIOR 2020)


There is an increased interest in solving complex constrained problems where part of the input is not given as facts, but received as raw sensor data such as images or speech. We will use ‘visual sudoku’ as a prototype problem, where the given cell digits are handwritten and provided as an image thereof. In this case, one first has to train and use a classifier to label the images, so that the labels can be used for solving the problem. In this paper, we explore the hybridisation of classifying the images with the reasoning of a constraint solver. We show that pure constraint reasoning on predictions does not give satisfactory results. Instead, we explore the possibilities of a tighter integration, by exposing the probabilistic estimates of the classifier to the constraint solver. This allows joint inference on these probabilistic estimates, where we use the solver to find the maximum likelihood solution. We explore the trade-off between the power of the classifier and the power of the constraint reasoning, as well as further integration through the additional use of structural knowledge. Furthermore, we investigate the effect of calibration of the probabilistic estimates on the reasoning. Our results show that such hybrid approaches vastly outperform a separate approach, which encourages a further integration of prediction (probabilities) and constraint solving.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. Amos, B., Kolter, J.Z.: Optnet: differentiable optimization as a layer in neural networks. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 136–145. (2017)

    Google Scholar 

  2. Bengio, Y.: Using a financial training criterion rather than a prediction criterion. Int. J. Neural Syst. 8(04), 433–443 (1997)

    Article  Google Scholar 

  3. Chen, L., Feng, Y., Huang, S., Luo, B., Zhao, D.: Encoding implicit relation requirements for relation extraction: a joint inference approach. Artif. Intell. 265, 45–66 (2018)

    Article  MathSciNet  Google Scholar 

  4. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  5. Guns, T., Stuckey, P.J., Tack, G.: Solution dominance over constraint satisfaction problems. CoRR abs/1812.09207 (2018)

    Google Scholar 

  6. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks (2017)

    Google Scholar 

  7. Ifrim, G., O’Sullivan, B., Simonis, H.: Properties of energy-price forecasts for scheduling. In: Milano, M. (ed.) CP 2012. LNCS, pp. 957–972. Springer, Heidelberg (2012).

    Chapter  Google Scholar 

  8. Kool, W., van Hoof, H., Welling, M.: Attention, learn to solve routing problems! In: ICLR 2019: 7th International Conference on Learning Representations (2019)

    Google Scholar 

  9. Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J.: Building machines that learn and think like people. Behav. Brain Sci. 40 (2017)

    Google Scholar 

  10. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  11. Li, Q., Anzaroot, S., Lin, W.P., Li, X., Ji, H.: Joint inference for cross-document information extraction. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, pp. 2225–2228. ACM (2011)

    Google Scholar 

  12. Li, Q., Ji, H., Huang, L.: Joint event extraction via structured prediction with global features. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 73–82 (2013)

    Google Scholar 

  13. Li, T., Gupta, V., Mehta, M., Srikumar, V.: A logic-driven framework for consistency of neural models. arXiv preprint arXiv:1909.00126 (2019)

  14. Mandi, J., Demirović, E., Stuckey, P., Guns, T., et al.: Smart predict-and-optimize for hard combinatorial optimization problems. arXiv preprint arXiv:1911.10092 (2019)

  15. Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., De Raedt, L.: Deepproblog: neural probabilistic logic programming. In: Advances in Neural Information Processing Systems, pp. 3749–3759 (2018)

    Google Scholar 

  16. Márquez-Neila, P., Salzmann, M., Fua, P.: Imposing hard constraints on deep networks: promises and limitations. arXiv preprint arXiv:1706.02025 (2017)

  17. Mukhopadhyay, A., Vorobeychik, Y., Dubey, A., Biswas, G.: Prioritized allocation of emergency responders based on a continuous-time incident prediction model. In: Adaptive Agents and Multi Agents Systems, pp. 168–177 (2017)

    Google Scholar 

  18. van den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. In: Advances in Neural Information Processing Systems, pp. 6306–6315 (2017)

    Google Scholar 

  19. Paszke, A., et al.: Automatic differentiation in PyTorch. In: NeurIPS Autodiff Workshop (2017)

    Google Scholar 

  20. Pathak, D., Krahenbuhl, P., Darrell, T.: Constrained convolutional neural networks for weakly supervised segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1796–1804 (2015)

    Google Scholar 

  21. Perron, L., team: Google’s or-tools

    Google Scholar 

  22. Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, pp. 61–74. MIT Press (1999)

    Google Scholar 

  23. Platt, J.C., Barr, A.H.: Constrained differential optimization. In: Neural Information Processing Systems, pp. 612–621 (1988)

    Google Scholar 

  24. Poon, H., Domingos, P.: Joint inference in information extraction. In: AAAI, vol. 7, pp. 913–918 (2007)

    Google Scholar 

  25. Punyakanok, V., Roth, D., Yih, W.T., Zimak, D.: Semantic role labeling via integer linear programming inference. In: Proceedings of the 20th International Conference on Computational Linguistics, COLING 2004, p. 1346. Association for Computational Linguistics, USA (2004)

    Google Scholar 

  26. Riedel, S.: Improving the accuracy and efficiency of map inference for Markov logic. arXiv preprint arXiv:1206.3282 (2012)

  27. Rossi, F., Van Beek, P., Walsh, T.: Handbook of Constraint Programming. Elsevier, Amsterdam (2006)

    MATH  Google Scholar 

  28. The High-Level Expert Group on Artificial Intelligence (AI HLEG): A definition of AI (2017)

    Google Scholar 

  29. Tulabandhula, T., Rudin, C.: Machine learning with operational costs. J. Mach. Learn. Res. 14(1), 1989–2028 (2013)

    MathSciNet  MATH  Google Scholar 

  30. Wang, P.W., Donti, P., Wilder, B., Kolter, Z.: Satnet: bridging deep learning and logical reasoning using a differentiable satisfiability solver. In: ICML 2019: Thirty-Sixth International Conference on Machine Learning, pp. 6545–6554 (2019)

    Google Scholar 

  31. Wang, Y., Chen, Q., Ahmed, M., Li, Z., Pan, W., Liu, H.: Joint inference for aspect-level sentiment analysis by deep neural networks and linguistic hints. IEEE Trans. Knowl. Data Eng. (2019)

    Google Scholar 

  32. Wilder, B.: Melding the data-decisions pipeline: decision-focused learning for combinatorial optimization. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (2019)

    Google Scholar 

Download references


This research received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

Author information

Authors and Affiliations


Corresponding authors

Correspondence to Maxime Mulamba , Jayanta Mandi , Rocsildes Canoy or Tias Guns .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mulamba, M., Mandi, J., Canoy, R., Guns, T. (2020). Hybrid Classification and Reasoning for Image-Based Constraint Solving. In: Hebrard, E., Musliu, N. (eds) Integration of Constraint Programming, Artificial Intelligence, and Operations Research. CPAIOR 2020. Lecture Notes in Computer Science(), vol 12296. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58941-7

  • Online ISBN: 978-3-030-58942-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics