Skip to main content
SpringerLink
Log in

Jigsaw puzzle solving techniques and applications: a survey

  • Survey
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

A jigsaw puzzle is a recreational activity that involves assembling a certain number of pieces into a combined and well-fitting unit without creating gaps between adjacent pieces. Two-dimensional puzzles are divided into two main categories, the “apictorial” in which the only information available is the shape of the pieces and the “pictorial” which may take into account not only the shape of the pieces, but also their content. Jigsaw puzzles are considered as one of the most popular category of puzzles. The majority of them are accompanied by a guiding image and there is only one “counterpart” for each side of each piece (pictorial jigsaw puzzles), although some more difficult variants have blank pieces, the so-called apictorial jigsaw puzzles. In this paper, we will examine the open problem of solving pictorial and apictorial jigsaw puzzles, and their various applications, such as the reconstruction of two-dimensional fragmented objects, the restoration of fragmented wall-paintings and the repair of shredded documents. We will also present an evaluation of the state-of-the- art jigsaw puzzle reassembly techniques in pictorial and apictorial puzzles.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Freeman, H., Garder, L.: Apictorial jigsaw puzzles: the computer solution of a problem in pattern recognition. IEEE Trans. Electron. Comput. 2, 118–127 (1964)

    Article  Google Scholar 

  2. Demaine, E.D., Demaine, M.L.: Jigsaw puzzles, edge matching, and polyomino packing: connections and complexity. Graphs Comb. 23(1), 195–208 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  3. Golomb, S.: Polyominoes, Patterns, Problems and Packing. Princeton University Press (1994)

  4. Kita, N., Miyata, K.: Computational design of polyomino puzzles. Vis. Comput. 37(4), 777–787 (2021)

    Article  Google Scholar 

  5. Zhang, M., Chen, S., Shu, Z., Xin, S.-Q., Zhao, J., Jin, G., Zhang, R., Beyerer, J.: Fast algorithm for 2d fragment assembly based on partial emd. Vis. Comput. 33(12), 1601–1612 (2017)

    Article  Google Scholar 

  6. Kleber, F., Sablatnig, R.: A survey of techniques for document and archaeology artefact reconstruction. In: 2009 10th International Conference on Document Analysis and Recognition, pp. 1061–1065. IEEE (2009)

  7. Radack, G.M., Badler, N.I.: Jigsaw puzzle matching using a boundary-centered polar encoding. Comput. Graphics Image Process. 19(1), 1–17 (1982)

    Article  Google Scholar 

  8. Hirota, K., Ohto, Y.: Image recognition in jigsaw puzzle assembly robot systems. Bull. Coll. Eng., Hosei Univ., Japan, pp. 87–93 (1986)

  9. Nagura, K., Sato, K., Maekawa, H., Morita, T., Fujii, K.: Partial contour processing using curvature function-assembly of jigsaw puzzle and recognition of moving figures. Syst. Comput. Jpn. 17(2), 30–39 (1986)

    Article  Google Scholar 

  10. Wolfson, H., Schonberg, E., Kalvin, A., Lamdan, Y.: Solving jigsaw puzzles by computer. Ann. Oper. Res. 12(1), 51–64 (1988)

    Article  MathSciNet  Google Scholar 

  11. Webster, R.W., LaFollette, P.S., Stafford, R.L.: Isthmus critical points for solving jigsaw puzzles in computer vision. IEEE Trans. Syst. Man Cybern. 21(5), 1271–1278 (1991)

    Article  Google Scholar 

  12. Schwartz, J.T., Sharir, M.: Identification of Partially Obscured Objects in Two Dimensions by Matching of Noisy ’characteristic Curve’s’. New York University. Courant Institute of Mathematical Sciences (1985)

  13. Goldberg, D., Malon, C., Bern, M.: A global approach to automatic solution of jigsaw puzzles. In: Proceedings of the Eighteenth Annual Symposium on Computational Geometry, pp. 82–87 (2002)

  14. Hoff, D.J., Olver, P.J.: Automatic solution of jigsaw puzzles. J. Math. Imaging Vis. 49(1), 234–250 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  15. Hoff, D.J., Olver, P.J.: Extensions of invariant signatures for object recognition. J. Math. Imaging Vis. 45(2), 176–185 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  16. Harel, P., Ben-Shahar, O.: Lazy caterer jigsaw puzzles: Models, properties, and a mechanical system-based solver. Preprint arXiv:2008.07644 (2020)

  17. De Bock, J., De Smet, R., Philips, W., D’Haeyer, J.: Constructing the topological solution of jigsaw puzzles. In: 2004 International Conference on Image Processing, 2004. ICIP’04., vol. 3, pp. 2127–2130. IEEE (2004)

  18. Kong, W., Kimia, B.B.: On solving 2d and 3d puzzles using curve matching. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 2, IEEE (2001)

  19. Zhu, L., Zhou, Z., Zhang, J., Hu, D.: A partial curve matching method for automatic reassembly of 2d fragments. In: Intelligent Computing in Signal Processing and Pattern Recognition, pp. 645–650. Springer (2006)

  20. Lalitha, K., Das, S., Menon, A., Varghese, K.: Graph-based clustering for apictorial jigsaw puzzles of hand shredded content-less pages. In: International Conference on Intelligent Human Computer Interaction, pp. 135–147. Springer (2016)

  21. da Gama Leitao, H.C., Stolfi, J.: A multiscale method for the reassembly of two-dimensional fragmented objects. IEEE Trans. Pattern Anal. Mach. Intell. 24(9), 1239–1251 (2002)

    Article  Google Scholar 

  22. Panagopoulos, T., Papaodysseus, C., Exarhos, M., Alexiou, C., Roussopoulos, G.: Automated reconstruction of fragmented, 1600 bc wall paintings (2002)

  23. McBride, J.C., Kimia, B.B.: Archaeological fragment reconstruction using curve-matching. In: 2003 Conference on Computer Vision and Pattern Recognition Workshop, vol. 1, pp. 3–3. IEEE (2003)

  24. Shin, H., Doumas, C., Funkhouser, T.A., Rusinkiewicz, S., Steiglitz, K., Vlachopoulos, A., Weyrich, T.: Analyzing fracture patterns in theranwall paintings. In: VAST, pp. 71–78. Citeseer (2010)

  25. Funkhouser, T., Shin, H., Toler-Franklin, C., Castañeda, A.G., Brown, B., Dobkin, D., Rusinkiewicz, S., Weyrich, T.: Learning how to match fresco fragments. J. Comput. Cult. Heritage (JOCCH) 4(2), 1–13 (2011)

    Article  Google Scholar 

  26. Naiman, A.E., Farber, E., Stein, Y.: Physical match. Informatica 43(2) (2019)

  27. Sizikova, E., Funkhouser, T.: Wall painting reconstruction using a genetic algorithm. J. Comput. Cult. Heritage (JOCCH) 11(1), 1–17 (2017)

    Google Scholar 

  28. Montusiewicz, J., Skulimowski, S.: A search method for reassembling the elements of a broken 2d object. Adv. Sci. Technol. Res. J. 14(3) (2020)

  29. Kosiba, D.A., Devaux, P.M., Balasubramanian, S., Gandhi, T.L., Kasturi, K.: An automatic jigsaw puzzle solver. In: Proceedings of 12th International Conference on Pattern Recognition, vol. 1, pp. 616–618. IEEE (1994)

  30. Chung, M.G., Fleck, M.M., Forsyth, D.A.: Jigsaw puzzle solver using shape and color. In: ICSP’98. 1998 Fourth International Conference on Signal Processing (Cat. No. 98TH8344), vol. 2, pp. 877–880. IEEE (1998)

  31. Yao, F.-H., Shao, G.-F.: A shape and image merging technique to solve jigsaw puzzles. Pattern Recogn. Lett. 24(12), 1819–1835 (2003)

    Article  Google Scholar 

  32. Makridis, M., Papamarkos, N.: A new technique for solving puzzles. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 40(3), 789–797 (2009)

    Article  Google Scholar 

  33. Nielsen, T.R., Drewsen, P., Hansen, K.: Solving jigsaw puzzles using image features. Pattern Recogn. Lett. 29(14), 1924–1933 (2008)

    Article  Google Scholar 

  34. Shih, H.-C., Lu, C.-L.: Divide-and-conquer jigsaw puzzle solving. In: 2018 IEEE Visual Communications and Image Processing (VCIP), pp. 1–2. IEEE (2018)

  35. Shen, B., Zhang, W., Zhao, H., Jin, Z., Wu, Y.: Solving pictorial jigsaw puzzle by stigmergy-inspired internet-based human collective intelligence. Preprint arXiv:1812.02559 (2018)

  36. Fornasier, M., Toniolo, D.: Fast, robust and efficient 2d pattern recognition for re-assembling fragmented images. Pattern Recogn. 38(11), 2074–2087 (2005)

    Article  Google Scholar 

  37. Papaodysseus, C., Exarhos, M., Panagopoulos, M., Rousopoulos, P., Triantafillou, C., Panagopoulos, T.: Image and pattern analysis of 1650 bc wall paintings and reconstruction. IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 38(4), 958–965 (2008)

    Article  Google Scholar 

  38. Tsamoura, E., Pitas, I.: Automatic color based reassembly of fragmented images and paintings. IEEE Trans. Image Process. 19(3), 680–690 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  39. Toler-Franklin, C., Brown, B., Weyrich, T., Funkhouser, T., Rusinkiewicz, S.: Multi-feature matching of fresco fragments. ACM Trans. Graph. (TOG) 29(6), 1–12 (2010)

    Article  Google Scholar 

  40. Derech, N., Tal, A., Shimshoni, I.: Solving archaeological puzzles. Pattern Recognition, 108065 (2021)

  41. Cantoni, V., Mosconi, M., Alessandra, S.: Javastylosis: a tool for computer-assisted chromatic and semantics based anastylosis of frescoes. In: Proceedings of the 21st International Conference on Computer Systems and Technologies’ 20, pp. 208–214 (2020)

  42. Liu, H., Cao, S., Yan, S.: Automated assembly of shredded pieces from multiple photos. IEEE Trans. Multimedia 13(5), 1154–1162 (2011)

    Article  Google Scholar 

  43. Richter, F., Ries, C.X., Cebron, N., Lienhart, R.: Learning to reassemble shredded documents. IEEE Trans. Multimedia 15(3), 582–593 (2012)

    Article  Google Scholar 

  44. Zhang, K., Li, X.: A graph-based optimization algorithm for fragmented image reassembly. Graph. Models 76(5), 484–495 (2014)

    Article  Google Scholar 

  45. Toyama, F., Fujiki, Y., Shoji, K., Miyamichi, J.: Assembly of puzzles using a genetic algorithm. In: Object Recognition Supported by User Interaction for Service Robots, vol. 4, pp. 389–392. IEEE (2002)

  46. Alajlan, N.: Solving square jigsaw puzzles using dynamic programming and the hungarian procedure. Am. J. Appl. Sci. 6(11), 1941 (2009)

    Article  Google Scholar 

  47. Cho, T.S., Avidan, S., Freeman, W.T.: A probabilistic image jigsaw puzzle solver. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 183–190. IEEE (2010)

  48. Pomeranz, D., Shemesh, M., Ben-Shahar, O.: A fully automated greedy square jigsaw puzzle solver. In: CVPR, pp. 9–16. IEEE (2011)

  49. Gallagher, A.C.: Jigsaw puzzles with pieces of unknown orientation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 382–389 (2012)

  50. Yu, R., Russell, C., Agapito, L.: Solving jigsaw puzzles with linear programming. Preprint arXiv:1511.04472 (2015)

  51. Paikin, G., Tal, A.: Solving multiple square jigsaw puzzles with missing pieces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4832–4839 (2015)

  52. Sholomon, D., David, O.E., Netanyahu, N.S.: An automatic solver for very large jigsaw puzzles using genetic algorithms. Genet. Program Evolvable Mach. 17(3), 291–313 (2016)

    Article  Google Scholar 

  53. Andalo, F.A., Taubin, G., Goldenstein, S.: Psqp: Puzzle solving by quadratic programming. IEEE Trans. Pattern Anal. Mach. Intell. 39(2), 385–396 (2016)

    Article  Google Scholar 

  54. Son, K., Hays, J., Cooper, D.B.: Solving square jigsaw puzzle by hierarchical loop constraints. IEEE Trans. Pattern Anal. Mach. Intell. 41(9), 2222–2235 (2018)

    Article  Google Scholar 

  55. Huroyan, V., Lerman, G., Wu, H.-T.: Solving jigsaw puzzles by the graph connection laplacian. SIAM J. Imag. Sci. 13(4), 1717–1753 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  56. Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: European Conference on Computer Vision, pp. 69–84. Springer (2016)

  57. Wei, C., Xie, L., Ren, X., Xia, Y., Su, C., Liu, J., Tian, Q., Yuille, A.L.: Iterative reorganization with weak spatial constraints: Solving arbitrary jigsaw puzzles for unsupervised representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1910–1919 (2019)

  58. Ostertag, C., Beurton-Aimar, M.: Matching ostraca fragments using a siamese neural network. Pattern Recogn. Lett. 131, 336–340 (2020)

    Article  Google Scholar 

  59. Dery, L., Mengistu, R., Awe, O.: Neural combinatorial optimization for solving jigsaw puzzles: A step towards unsupervised pre-training. Stanford Univ., Stanford, CA, USA, Tech. Rep (2017)

  60. Paumard, M.-M., Picard, D., Tabia, H.: Jigsaw puzzle solving using local feature co-occurrences in deep neural networks. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 1018–1022, IEEE (2018)

  61. Paumard, M.-M., Picard, D., Tabia, H.: Deepzzle: solving visual jigsaw puzzles with deep learning and shortest path optimization. IEEE Trans. Image Process. 29, 3569–3581 (2020)

    Article  MATH  Google Scholar 

  62. Li, R., Liu, S., Wang, G., Liu, G., Zeng, B.: Jigsawgan: auxiliary learning for solving jigsaw puzzles with generative adversarial networks. IEEE Trans. Image Process. 31, 513–524 (2021)

    Article  Google Scholar 

  63. Kwon, H., Yoon, H., Park, K.-W.: Captcha image generation: two-step style-transfer learning in deep neural networks. Sensors 20(5), 1495 (2020)

    Article  Google Scholar 

  64. Kumar, A., Singh, A.P.: Contour based deep learning engine to solve captcha. In: 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS), vol. 1, pp. 723–727. IEEE (2021)

  65. Elson, J., Douceur, J.R., Howell, J., Saul, J.: Asirra: a captcha that exploits interest-aligned manual image categorization. CCS 7, 366–374 (2007)

    Google Scholar 

  66. Payal, N., Challa, R.K.: Ajigjax: a hybrid image based model for captcha/carp. In: 2016 IEEE Uttar Pradesh Section International Conference on Electrical, Computer and Electronics Engineering (UPCON), pp. 38–43. IEEE (2016)

  67. Sauer, G., Hochheiser, H., Feng, J., Lazar, J.: Towards a universally usable captcha. In: Proceedings of the 4th Symposium on Usable Privacy and Security, vol. 6, p. 1 (2008)

  68. Gao, H., Yao, D., Liu, H., Liu, X., Wang, L.: A novel image based captcha using jigsaw puzzle. In: 2010 13th IEEE International Conference on Computational Science and Engineering, pp. 351–356. IEEE (2010)

  69. Ali, F.A.B.H., Karim, F.B.: Development of captcha system based on puzzle. In: 2014 International Conference on Computer, Communications, and Control Technology (I4CT), pp. 426–428. IEEE (2014)

  70. Mondal, D., Wang, Y., Durocher, S.: Robust solvers for square jigsaw puzzles. In: 2013 International Conference on Computer and Robot Vision, pp. 249–256. IEEE (2013)

  71. Sholomon, D., David, O.E., Netanyahu, N.S.: Dnn-buddies: A deep neural network-based estimation metric for the jigsaw puzzle problem. In: International Conference on Artificial Neural Networks, pp. 170–178. Springer (2016)

  72. Paumard, M.-M., Picard, D., Tabia, H.: Image reassembly combining deep learning and shortest path problem. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 153–167 (2018)

  73. Bunke, H., Kaufmann, G.: Jigsaw puzzle solving using approximate string matching and best-first search. In: International Conference on Computer Analysis of Images and Patterns, pp. 299–308. Springer (1993)

  74. Li, D., Yang, Y., Song, Y.-Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5542–5550 (2017)

  75. Garcin, M., Le Cozannet, G.: The driving factors of coastal evolution: toward a systemic approach. In: Climate Change and Sea Level Rise; Coastal Vulnerability and Societal Impacts (2013)

  76. Wegener, A.: Die entstehung der kontinente. Geol. Rundsch. 3(4), 276–292 (1912)

    Article  Google Scholar 

  77. Mascret, A., Devogele, T., Berre, I.L., Henaff, A.: Coastline matching process based on the discrete frechet distance. In: Progress in Spatial Data Handling, pp. 383–400. Springer (2006)

Download references

Acknowledgements

This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH - CREATE - INNOVATE B cycle (Project Code: T2EDK-03135).

Author information

Authors and Affiliations

  1. Department of Management Science and Technology, Hellenic Mediterranean University, 72100, Agios Nikolaos, Greece

    Smaragda Markaki & Costas Panagiotakis

Authors
  1. Smaragda Markaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Costas Panagiotakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Smaragda Markaki.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Markaki, S., Panagiotakis, C. Jigsaw puzzle solving techniques and applications: a survey. Vis Comput 39, 4405–4421 (2023). https://doi.org/10.1007/s00371-022-02598-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02598-9

Keywords

Search

Navigation