Skip to main content
SpringerLink
Log in

Evaginating scientific charts: Recovering direct and derived information encodings from chart images

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

Recovering information encodings from charts is to extract textual and graphical information from the diverse representations of charts to aid various applications that entail the hidden and directly available information. This paper proposes an end-to-end relationship-aware model for evaginating scientific charts for which an automated encoder-decoder framework is adopted. The framework integrates the CNN-LSTM model to extract direct and derived data by the influence of semantic relationships between the textual and graphical components. Semantic relationship makes the model robust towards diverse chart structures and orientations. Entity relationship-aware module extracts and builds relations amid textual-graphical components of the charts, and decodes the hidden data from the chart images. Our framework is one of its kind to recover data encoding from chart images based upon their inter-object semantic relationships, to the best of our knowledge. Model is tested upon public datasets, obtaining more than 97% accuracy compared with benchmark systems.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Al-Zaidy RA, Choudhury SR, Giles CL (2019) Automatic summary generation for scientific data charts. In: Workshops at the thirtieth AAAI conference on artificial intelligence

  • Blue Leaf Software - Dagra (2020) [Online]. Available: https://blueleafsoftware.com/

  • Casado-García Á, Domínguez C, García-Domínguez M, Heras J, Inés A, Mata E, Pascual V (2019) CLoDSA: a tool for augmentation in classification, localization, detection, semantic segmentation and instance segmentation tasks. BMC Bioinform 20(1):323

    Article  Google Scholar 

  • Chen Z, Wang Y, Wang Q, Wang Y, Huamin Qu (2019) Towards automated infographic design: deep learning-based auto-extraction of extensible timeline. IEEE Trans Visual Comput Graphics 26(1):917–926

    Google Scholar 

  • Chen Z, Cafarella M, Adar E (2015) Diagramflyer: A search engine for data-driven diagrams. In: Proceedings of the 24th International conference on world wide web, pp. 183-186

  • Choi J, Jung S, Park DG, Choo J, Elmqvist N (2019) Visualizing for the non-visual: enabling the visually impaired to use visualization. Comput Graph Forum 38(3):249–260

    Article  Google Scholar 

  • Choudhury SR, Wang S, Mitra P, Giles CL (2015) Automated data extraction from scholarly line graphs. In: GREC

  • Cliche M, Rosenberg D, Madeka D, Yee C (2017) Scatteract: Automated extraction of data from scatter plots. Joint European conference on machine learning and knowledge discovery in databases. Springer, Cham, pp 135–150

    Chapter  Google Scholar 

  • Dai W, Wang M, Niu Z, Zhang J (2018) Chart decoder: Generating textual and numeric information from chart images automatically. J vis Lang Comput 48:101–109

    Article  Google Scholar 

  • DataThief (2006) [Online]. Available: https://datathief.org/

  • Deshpande AP, Mahender CN (2020) Summarization of graph using question answer approach. Information and communication technology for sustainable development. Springer, Singapore, pp 205–216

    Chapter  Google Scholar 

  • Engauge Digitizer (2018) [Online]. Available: http://markummitchell.github.io/engauge-digitizer/

  • Elzer S, Carberry S, Zukerman I (2011) The automated understanding of simple bar charts. Artif Intell 175(2):526–555

    Article  MathSciNet  Google Scholar 

  • Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vision 88(2):303–338

    Article  Google Scholar 

  • Fu TJ, Li PH, Ma WY (2019) GraphRel: modeling text as relational graphs for joint entity and relation extraction. In: Proceedings of the 57th Annual meeting of the association for computational linguistics, pp. 1409-1418

  • Fu J, Zhu B, Cui W, Ge S, Wang Y, Zhang H, Huang H, Tang Y, Zhang D, Ma X (2020) Chartem: reviving chart images with data embedding. IEEE Trans Visual Comput Graph 27(2):337–346

    Article  Google Scholar 

  • Gao J, Carrillo RE, Barner KE (2010) Image categorization for improving accessibility to information graphics. In: Proceedings of the 12th International ACM SIGACCESS Conference on computers and accessibility, pp. 265-266

  • Gupta A, Vedaldi A, Zisserman A (2016) Synthetic data for text localisation in natural images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2315-2324

  • Harper J, Agrawala M (2014) Deconstructing and restyling D3 visualizations. In: Proceedings of the 27th Annual ACM Symposium on user interface software and technology, pp. 253-262

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer vision and pattern recognition, pp. 770-778

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  • Huang W, Tan CL (2007) A system for understanding imaged infographics and its applications. In: Proceedings of the 2007 ACM Symposium on document engineering, pp. 9-18

  • Jayant C, Renzelmann M, Wen D, Krisnandi S, Ladner R, Comden D (2007) Automated tactile graphics translation: in the field. In: Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 75-82

  • Jung D, Kim W, Song H, Hwang JI, Lee B, Kim B, Seo J (2017) Chartsense: Interactive data extraction from chart images. In: Proceedings of the 2017 chi conference on human factors in computing systems, pp. 6706-6717

  • Kaggle Four Shapes Dataset (2017) Available Online: https://www.kaggle.com/smeschke/four-shapes

  • Kafle K, Shrestha R, Cohen S, Price B, Kanan C (2020) Answering questions about data visualizations using efficient bimodal fusion. In: Proceedings of the IEEE/CVF Winter conference on applications of computer vision, pp. 1498-1507

  • Kahou SE, Michalski V, Atkinson A, Kádár Á, Trischler A, Bengio Y (2017) FigureQA: an annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300

  • Kataria S, Browuer W, Mitra P, Giles CL (2008) Automatic extraction of data points and text blocks from 2-dimensional plots in digital documents. AAAI 8:1169–1174

    Google Scholar 

  • Li Z, Carberry S, Fang H, McCoy KF, Peterson K, Stagitis M (2015) A novel methodology for retrieving infographics utilizing structure and message content. Data Knowl Eng 100:191–210

    Article  Google Scholar 

  • Li K, Zhang Y, Li K, Li Y, Fu Y (2019) Visual semantic reasoning for image-text matching. In: Proceedings of the IEEE/CVF International conference on computer vision, pp. 4654-4662

  • Liu X, Klabjan D, NBless P (2019) Data extraction from charts via single deep neural network. arXiv preprint arXiv:1906.11906

  • Mayhua A, Gomez-Nieto E, Heer J, Poco J (2018) Extracting visual encodings from map chart images with color-encoded scalar values. In: 2018 31st SIBGRAPI Conference on graphics, patterns and images (SIBGRAPI), pp. 142-149. IEEE.

  • McAuley J, Rohan G, Tamara M (2019) ExploroBOT: rapid exploration with chart automation. In VISIGRAPP (3: IVAPP), pp. 225–232

  • Nagy R, Dicker A, Meyer-Wegener K (2011) NEOCR: A configurable dataset for natural image text recognition. In: International workshop on camera-based document analysis and recognition. Springer, Berlin, Heidelberg, pp. 150-163

  • Poco J, Heer J (2017) Reverse-engineering visualizations: recovering visual encodings from chart images. In Computer Graphics Forum 36(3):353–363

    Article  Google Scholar 

  • Poco J, Mayhua A, Heer J (2017) Extracting and retargeting color mappings from bitmap images of visualizations. IEEE Trans Visual Comput Graphics 24(1):637–646

    Article  Google Scholar 

  • Plot Digitizer (2015) [Online]. Available: http://plotdigitizer.sourceforge.net

  • Ren S, He K, Girshick R, Sun J (2016) Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  • Riesen K (2015) Structural pattern recognition with graph edit distance. Advances in computer vision and pattern recognition. Springer, Cham, pp 1–164

    Google Scholar 

  • Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J (2016) Vega-lite: a grammar of interactive graphics. IEEE Trans Visual Comput Graph 23(1):341–350

    Article  Google Scholar 

  • Savva M, Kong N, Chhajta A, Fei-Fei L, Agrawala M, Heer J (2011) Revision: Automated classification, analysis and redesign of chart images. In: Proceedings of the 24th Annual ACM Symposium on user interface software and technology, pp. 393-402

  • Veit A, Matera T, Neumann L, Matas J, Belongie S (2016) Coco-text: dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140

  • Yu H, Li H, Mao D, Cai Q (2020) A relationship extraction method for domain knowledge graph construction. World Wide Web 23(2):735–753

    Article  Google Scholar 

  • Zhang F, Luan J, Xu Z, Chen W (2020) DetReco: object-Text detection and recognition based on deep neural network. Math Problems Eng. https://doi.org/10.1155/2020/2365076

    Article  Google Scholar 

  • Zhou YP, Tan CL (2001) Learning-based scientific chart recognition. In: 4th IAPR International workshop on graphics recognition, GREC. pp. 482-492

  • Zhou F, Zhao Y, Chen W, Tan Y, Xu Y, Chen Y, Liu C, Zhao Y (2021) Reverse-engineering bar charts using neural networks. J Visual 24(2):419–435

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of CSE, DSPM-IIITNR, Naya Raipur, India

    Prerna Mishra & Santosh Kumar

  2. Department of Mathematics, DSPM-IIITNR, Naya Raipur, India

    Mithilesh Kumar Chaube

Authors
  1. Prerna Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Santosh Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mithilesh Kumar Chaube
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Prerna Mishra.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mishra, P., Kumar, S. & Chaube, M.K. Evaginating scientific charts: Recovering direct and derived information encodings from chart images. J Vis 25, 343–359 (2022). https://doi.org/10.1007/s12650-021-00800-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-021-00800-z

Keywords

Search

Navigation