Skip to main content

Evaginating scientific charts: Recovering direct and derived information encodings from chart images


Recovering information encodings from charts is to extract textual and graphical information from the diverse representations of charts to aid various applications that entail the hidden and directly available information. This paper proposes an end-to-end relationship-aware model for evaginating scientific charts for which an automated encoder-decoder framework is adopted. The framework integrates the CNN-LSTM model to extract direct and derived data by the influence of semantic relationships between the textual and graphical components. Semantic relationship makes the model robust towards diverse chart structures and orientations. Entity relationship-aware module extracts and builds relations amid textual-graphical components of the charts, and decodes the hidden data from the chart images. Our framework is one of its kind to recover data encoding from chart images based upon their inter-object semantic relationships, to the best of our knowledge. Model is tested upon public datasets, obtaining more than 97% accuracy compared with benchmark systems.

Graphical abstract

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11


  1. Al-Zaidy RA, Choudhury SR, Giles CL (2019) Automatic summary generation for scientific data charts. In: Workshops at the thirtieth AAAI conference on artificial intelligence

  2. Blue Leaf Software - Dagra (2020) [Online]. Available:

  3. Casado-García Á, Domínguez C, García-Domínguez M, Heras J, Inés A, Mata E, Pascual V (2019) CLoDSA: a tool for augmentation in classification, localization, detection, semantic segmentation and instance segmentation tasks. BMC Bioinform 20(1):323

    Article  Google Scholar 

  4. Chen Z, Wang Y, Wang Q, Wang Y, Huamin Qu (2019) Towards automated infographic design: deep learning-based auto-extraction of extensible timeline. IEEE Trans Visual Comput Graphics 26(1):917–926

    Google Scholar 

  5. Chen Z, Cafarella M, Adar E (2015) Diagramflyer: A search engine for data-driven diagrams. In: Proceedings of the 24th International conference on world wide web, pp. 183-186

  6. Choi J, Jung S, Park DG, Choo J, Elmqvist N (2019) Visualizing for the non-visual: enabling the visually impaired to use visualization. Comput Graph Forum 38(3):249–260

    Article  Google Scholar 

  7. Choudhury SR, Wang S, Mitra P, Giles CL (2015) Automated data extraction from scholarly line graphs. In: GREC

  8. Cliche M, Rosenberg D, Madeka D, Yee C (2017) Scatteract: Automated extraction of data from scatter plots. Joint European conference on machine learning and knowledge discovery in databases. Springer, Cham, pp 135–150

    Chapter  Google Scholar 

  9. Dai W, Wang M, Niu Z, Zhang J (2018) Chart decoder: Generating textual and numeric information from chart images automatically. J vis Lang Comput 48:101–109

    Article  Google Scholar 

  10. DataThief (2006) [Online]. Available:

  11. Deshpande AP, Mahender CN (2020) Summarization of graph using question answer approach. Information and communication technology for sustainable development. Springer, Singapore, pp 205–216

    Chapter  Google Scholar 

  12. Engauge Digitizer (2018) [Online]. Available:

  13. Elzer S, Carberry S, Zukerman I (2011) The automated understanding of simple bar charts. Artif Intell 175(2):526–555

    MathSciNet  Article  Google Scholar 

  14. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vision 88(2):303–338

    Article  Google Scholar 

  15. Fu TJ, Li PH, Ma WY (2019) GraphRel: modeling text as relational graphs for joint entity and relation extraction. In: Proceedings of the 57th Annual meeting of the association for computational linguistics, pp. 1409-1418

  16. Fu J, Zhu B, Cui W, Ge S, Wang Y, Zhang H, Huang H, Tang Y, Zhang D, Ma X (2020) Chartem: reviving chart images with data embedding. IEEE Trans Visual Comput Graph 27(2):337–346

    Article  Google Scholar 

  17. Gao J, Carrillo RE, Barner KE (2010) Image categorization for improving accessibility to information graphics. In: Proceedings of the 12th International ACM SIGACCESS Conference on computers and accessibility, pp. 265-266

  18. Gupta A, Vedaldi A, Zisserman A (2016) Synthetic data for text localisation in natural images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2315-2324

  19. Harper J, Agrawala M (2014) Deconstructing and restyling D3 visualizations. In: Proceedings of the 27th Annual ACM Symposium on user interface software and technology, pp. 253-262

  20. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer vision and pattern recognition, pp. 770-778

  21. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  22. Huang W, Tan CL (2007) A system for understanding imaged infographics and its applications. In: Proceedings of the 2007 ACM Symposium on document engineering, pp. 9-18

  23. Jayant C, Renzelmann M, Wen D, Krisnandi S, Ladner R, Comden D (2007) Automated tactile graphics translation: in the field. In: Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 75-82

  24. Jung D, Kim W, Song H, Hwang JI, Lee B, Kim B, Seo J (2017) Chartsense: Interactive data extraction from chart images. In: Proceedings of the 2017 chi conference on human factors in computing systems, pp. 6706-6717

  25. Kaggle Four Shapes Dataset (2017) Available Online:

  26. Kafle K, Shrestha R, Cohen S, Price B, Kanan C (2020) Answering questions about data visualizations using efficient bimodal fusion. In: Proceedings of the IEEE/CVF Winter conference on applications of computer vision, pp. 1498-1507

  27. Kahou SE, Michalski V, Atkinson A, Kádár Á, Trischler A, Bengio Y (2017) FigureQA: an annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300

  28. Kataria S, Browuer W, Mitra P, Giles CL (2008) Automatic extraction of data points and text blocks from 2-dimensional plots in digital documents. AAAI 8:1169–1174

    Google Scholar 

  29. Li Z, Carberry S, Fang H, McCoy KF, Peterson K, Stagitis M (2015) A novel methodology for retrieving infographics utilizing structure and message content. Data Knowl Eng 100:191–210

    Article  Google Scholar 

  30. Li K, Zhang Y, Li K, Li Y, Fu Y (2019) Visual semantic reasoning for image-text matching. In: Proceedings of the IEEE/CVF International conference on computer vision, pp. 4654-4662

  31. Liu X, Klabjan D, NBless P (2019) Data extraction from charts via single deep neural network. arXiv preprint arXiv:1906.11906

  32. Mayhua A, Gomez-Nieto E, Heer J, Poco J (2018) Extracting visual encodings from map chart images with color-encoded scalar values. In: 2018 31st SIBGRAPI Conference on graphics, patterns and images (SIBGRAPI), pp. 142-149. IEEE.

  33. McAuley J, Rohan G, Tamara M (2019) ExploroBOT: rapid exploration with chart automation. In VISIGRAPP (3: IVAPP), pp. 225–232

  34. Nagy R, Dicker A, Meyer-Wegener K (2011) NEOCR: A configurable dataset for natural image text recognition. In: International workshop on camera-based document analysis and recognition. Springer, Berlin, Heidelberg, pp. 150-163

  35. Poco J, Heer J (2017) Reverse-engineering visualizations: recovering visual encodings from chart images. In Computer Graphics Forum 36(3):353–363

    Article  Google Scholar 

  36. Poco J, Mayhua A, Heer J (2017) Extracting and retargeting color mappings from bitmap images of visualizations. IEEE Trans Visual Comput Graphics 24(1):637–646

    Article  Google Scholar 

  37. Plot Digitizer (2015) [Online]. Available:

  38. Ren S, He K, Girshick R, Sun J (2016) Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  39. Riesen K (2015) Structural pattern recognition with graph edit distance. Advances in computer vision and pattern recognition. Springer, Cham, pp 1–164

    Google Scholar 

  40. Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J (2016) Vega-lite: a grammar of interactive graphics. IEEE Trans Visual Comput Graph 23(1):341–350

    Article  Google Scholar 

  41. Savva M, Kong N, Chhajta A, Fei-Fei L, Agrawala M, Heer J (2011) Revision: Automated classification, analysis and redesign of chart images. In: Proceedings of the 24th Annual ACM Symposium on user interface software and technology, pp. 393-402

  42. Veit A, Matera T, Neumann L, Matas J, Belongie S (2016) Coco-text: dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140

  43. Yu H, Li H, Mao D, Cai Q (2020) A relationship extraction method for domain knowledge graph construction. World Wide Web 23(2):735–753

    Article  Google Scholar 

  44. Zhang F, Luan J, Xu Z, Chen W (2020) DetReco: object-Text detection and recognition based on deep neural network. Math Problems Eng.

    Article  Google Scholar 

  45. Zhou YP, Tan CL (2001) Learning-based scientific chart recognition. In: 4th IAPR International workshop on graphics recognition, GREC. pp. 482-492

  46. Zhou F, Zhao Y, Chen W, Tan Y, Xu Y, Chen Y, Liu C, Zhao Y (2021) Reverse-engineering bar charts using neural networks. J Visual 24(2):419–435

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Prerna Mishra.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mishra, P., Kumar, S. & Chaube, M.K. Evaginating scientific charts: Recovering direct and derived information encodings from chart images. J Vis (2021).

Download citation


  • Information extraction
  • Reverse engineering
  • Charts
  • Relation graphs
  • Direct and derived data