Skip to main content

Visual-Linguistic Methods for Receipt Field Recognition

Part of the Lecture Notes in Computer Science book series (LNIP,volume 11362)


Receipts are crucial for many businesses’ operation, where expenses are tracked meticulously. Receipt documents are often scanned into images, digitized and analyzed before the information is streamed into institutional financial applications. The precise extraction of expense data from receipt images is a difficult task owed to the high variance in fonts and layouts, the frailty of the print paper, unstructured scanning environments and an immeasurable amount of domains. We propose a method that combines visual and linguistic features for automatic information retrieval from receipt images using deep network architectures, which outperforms existing approaches. Our Skip-Rect Embedding (SRE) descriptor is demonstrated in two canonical applications for receipt information retrieval: field extraction and Optical Character Recognition (OCR) error enhancement.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-20890-5_35
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-20890-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.


  1. 1.

    Based on experiments we performed using the prolific Tesseract OCR engine on preprocessed, phone-captured images [19].


  1. Aslan, E., Karakaya, T., Unver, E., Akgül, Y.S.: A part based modeling approach for invoice parsing. In: VISIGRAPP (3: VISAPP), pp. 392–399 (2016)

    Google Scholar 

  2. Cao, K., Rei, M.: A joint model for word embedding and word morphology. arXiv preprint arXiv:1606.02601 (2016)

  3. Cesarini, F., Francesconi, E., Gori, M., Soda, G.: Analysis and understanding of multi-class invoices. Doc. Anal. Recogn. 6(2), 102–114 (2003)

    CrossRef  Google Scholar 

  4. Chen, K., Seuret, M., Liwicki, M., Hennebert, J., Ingold, R.: Page segmentation of historical document images with convolutional autoencoders. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1011–1015. IEEE (2015)

    Google Scholar 

  5. Hamza, H., Belaïd, Y., Belaïd, A.: Case-based reasoning for invoice analysis and recognition. In: Weber, R.O., Richter, M.M. (eds.) ICCBR 2007. LNCS (LNAI), vol. 4626, pp. 404–418. Springer, Heidelberg (2007).

    CrossRef  Google Scholar 

  6. Jaech, A., Mulcaire, G., Ostendorf, M., Smith, N.A.: A neural model for language identification in code-switched tweets. In: Proceedings of the Second Workshop on Computational Approaches to Code Switching, pp. 60–64 (2016)

    Google Scholar 

  7. Janssen, B., Saund, E., Bier, E., Wall, P., Sprague, M.A.: Receipts2Go: the big world of small documents. In: Proceedings of the 2012 ACM symposium on Document Engineering, pp. 121–124. ACM (2012)

    Google Scholar 

  8. Johnson, J., Karpathy, A., Fei-Fei, L.: DenseCap: fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4565–4574 (2016)

    Google Scholar 

  9. Kang, L., Kumar, J., Ye, P., Li, Y., Doermann, D.: Convolutional neural networks for document image classification. In: 2014 22nd International Conference on Pattern Recognition (ICPR), pp. 3168–3172. IEEE (2014)

    Google Scholar 

  10. Klein, B., Agne, S., Dengel, A.: Results of a study on invoice-reading systems in Germany. In: Marinai, S., Dengel, A.R. (eds.) DAS 2004. LNCS, vol. 3163, pp. 451–462. Springer, Heidelberg (2004).

    CrossRef  Google Scholar 

  11. Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual relationship detection with language priors. arXiv:1608.00187 [cs], July 2016

  12. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv:1301.3781 [cs, stat], January 2013

  13. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. arXiv:1310.4546 [cs, stat], October 2013

  14. Palm, R.B., Winther, O., Laws, F.: CloudScan-a configuration-free invoice analysis system using recurrent neural networks. arXiv preprint arXiv:1708.07403 (2017)

  15. Pennington, J., Socher, R., D. Manning, C.: GloVe: global vectors for word representation. (2014).

  16. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).

    CrossRef  Google Scholar 

  17. Santos, C.D., Zadrozny, B.: Learning character-level representations for part-of-speech tagging. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pp. 1818–1826 (2014)

    Google Scholar 

  18. Schulz, F., Ebbecke, M., Gillmann, M., Adrian, B., Agne, S., Dengel, A.: Seizing the treasure: transferring layout knowledge in invoice analysis. In: 10 International Conference on Document Analysis and Recognition (ICDAR), Barcelona, Spain (2009)

    Google Scholar 

  19. Smith, R.: An overview of the Tesseract OCR engine. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, ICDAR 2007, vol. 02, pp. 629–633. IEEE Computer Society, Washington, DC, USA (2007).

  20. Sorio, E., Bartoli, A., Davanzo, G., Medvet, E.: Open world classification of printed invoices. In: Proceedings of the 10th ACM symposium on Document engineering, pp. 187–190. ACM (2010)

    Google Scholar 

  21. Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway Networks. arXiv:1505.00387, May 2015

  22. Vosoughi, S., Vijayaraghavan, P., Roy, D.: Tweet2vec: learning Tweet embeddings using character-level CNN-LSTM encoder-decoder. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016, pp. 1041–1044. ACM, New York, NY, USA (2016).,

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Rinon Gal .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1348 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Gal, R., Morag, N., Shilkrot, R. (2019). Visual-Linguistic Methods for Receipt Field Recognition. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11362. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20889-9

  • Online ISBN: 978-3-030-20890-5

  • eBook Packages: Computer ScienceComputer Science (R0)