Image Reassembly Combining Deep Learning and Shortest Path Problem

  • Marie-Morgane Paumard
  • David Picard
  • Hedi Tabia
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11210)


This paper addresses the problem of reassembling images from disjointed fragments. More specifically, given an unordered set of fragments, we aim at reassembling one or several possibly incomplete images. The main contributions of this work are: (1) several deep neural architectures to predict the relative position of image fragments that outperform the previous state of the art; (2) casting the reassembly problem into the shortest path in a graph problem for which we provide several construction algorithms depending on available information; (3) a new dataset of images taken from the Metropolitan Museum of Art (MET) dedicated to image reassembly for which we provide a clear setup and a strong baseline.


Fragments reassembly Jigsaw puzzle Image classification Cultural heritage Deep learning 


  1. 1.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. NIPS 1, 1097–1105 (2012)Google Scholar
  2. 2.
    Johnson, J., Karpathy, A., Fei-Fei, L.: DenseCap: fully convolutional localization networks for dense captioning. In: CVPR (2016)Google Scholar
  3. 3.
    Gordo, A., Almazán, J., Revaud, J., Larlus, D.: Deep image retrieval: learning global representations for image search. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 241–257. Springer, Cham (2016). Scholar
  4. 4.
    Doersch, C., Gupta, A., Efros, A.: Unsupervised visual representation learning by context prediction. In: ICCV (2015)Google Scholar
  5. 5.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. IJCV 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Rasheed, N., Nordin, M.J.: A survey of classification and reconstruction methods for the 2D archaeological objects. In: ISTMET, pp. 142–147, August 2015Google Scholar
  7. 7.
    Rasheed, N., Nordin, M.J.: A survey of computer methods in reconstruction of 3D archaeological pottery objects. Int. J. Adv. Res. 3, 712–714 (2015)Google Scholar
  8. 8.
    McBride, J., Kimia, B.: Archaeological fragment reconstruction using curve-matching. In: CVPRW (2003)Google Scholar
  9. 9.
    Jampy, F., Hostein, A., Fauvet, E., Laligant, O., Truchetet, F.: 3D puzzle reconstruction for archeological fragments. In: 3DIPM (2015)Google Scholar
  10. 10.
    Zhu, L., Zhou, Z., Zhang, J., Hu, D.: A partial curve matching method for automatic reassembly of 2D fragments. In: Huang, D.S., Li, K., Irwin, G.W. (eds.) Intelligent Computing in Signal Processing and Pattern Recognition. LNCIS, vol. 345, pp. 645–650. Springer, Berlin (2006). Scholar
  11. 11.
    Hammoudeh, Z., Pollett, C.: Clustering-based, fully automated mixed-bag jigsaw puzzle solving. In: Felsberg, M., Heyden, A., Krüger, N. (eds.) CAIP 2017. LNCS, vol. 10425, pp. 205–217. Springer, Cham (2017). Scholar
  12. 12.
    Andaló, F., Taubin, G., Goldenstein, S.: PSQP: puzzle solving by quadratic programming. IEEE TPAMI 39, 385–396 (2017)CrossRefGoogle Scholar
  13. 13.
    Lifang, C., Cao, D., Liu, Y.: A new intelligent jigsaw puzzle algorithm base on mixed similarity and symbol matrix. IJPRAI 32, 1859001 (2018)MathSciNetGoogle Scholar
  14. 14.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS, pp. 3111–3119 (2013)Google Scholar
  15. 15.
    Kiros, R., et al.: Skip-thought vectors. In: NIPS, pp. 3294–3302 (2015)Google Scholar
  16. 16.
    Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles (2015)Google Scholar
  17. 17.
    Gur, S., Ben-Shahar, O.: From square pieces to brick walls: the next challenge in solving jigsaw puzzles. In: ICCV (2017)Google Scholar
  18. 18.
    Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear CNN models for fine-grained visual recognition. In: ICCV, pp. 1449–1457 (2015)Google Scholar
  19. 19.
    Gao, Y., Beijbom, O., Zhang, N., Darrell, T.: Compact bilinear pooling. In: IEEE CVPR, pp. 317–326 (2016)Google Scholar
  20. 20.
    Kim, J.H., On, K.W., Lim, W., Ha, J., Zhang, B.-T.: Hadamard product for low-rank bilinear pooling. In: ICLR (2017)Google Scholar
  21. 21.
    Ben-younes, H., Cadene, R., Cord, M., Thome, N.: MUTAN: multimodal tucker fusion for visual question answering, pp. 2612–2620 (2017)Google Scholar
  22. 22.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ILSVRC (2014)Google Scholar
  23. 23.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015)Google Scholar
  24. 24.
    Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines. In: ICML (2010)Google Scholar
  25. 25.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE CVPR, pp. 770–778 (2016)Google Scholar
  26. 26.
    Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
  27. 27.
    Dijkstra, E.: A note on two problems in connexion with graphs. Numerische Mathematik 1, 269–271 (1959)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Marie-Morgane Paumard
    • 1
  • David Picard
    • 1
    • 2
  • Hedi Tabia
    • 1
  1. 1.ETIS, UMR 8051, Université Paris Seine, Université Cergy-Pontoise, ENSEA, CNRSCergy-PontoiseFrance
  2. 2.Sorbonne Université, CNRS, Laboratoire d’Informatique de Paris 6ParisFrance

Personalised recommendations