Skip to main content
SpringerLink
Log in

Reverse-engineering bar charts using neural networks

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

Reverse-engineering bar charts extract textual and numeric information from the visual representations of bar charts to support application scenarios that require the underlying information. In this paper, we propose a neural network-based method for reverse-engineering bar charts. We adopt a neural network-based object detection model to simultaneously localize and classify textual information. This approach improves the efficiency of textual information extraction. We design an encoder-decoder framework that integrates convolutional and recurrent neural networks to extract numeric information. We further introduce an attention mechanism into the framework to achieve high accuracy and robustness. Synthetic and real-world datasets are used to evaluate the effectiveness of the method. To the best of our knowledge, this work takes the lead in constructing a complete neural network-based method of reverse-engineering bar charts.

Graphic abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Al-ZaidyRA, Giles CL (2015) Automatic extraction of data from bar charts. In: Proceedings of the 8th international conference on knowledge capture. ACM, p 30

  • Al-ZaidyRA, Choudhury SR, Giles CL (2016) Automatic summary generation for scientific data charts. In: Workshops at the 30th AAAI conference on artificial intelligence. AAAI, pp 658–663

  • BahdanauD, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  • Battle L, Duan P, Miranda Z, Mukusheva D, Chang R, Stonebraker M (2018) Beagle: automated extraction and interpretation of visualizations from the web. In: Proceedings of the 2018 CHI conference on human factors in computing systems. ACM, p 594

  • BeltramelliT (2018) pix2code: generating code from a graphical user interface screenshot. In: Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems. ACM, p 3

  • Bi C, Yang L, Duan Y, Shi Y (2019a) A survey on visualization of tensor field. J Vis 22(3):641–660

    Article  Google Scholar 

  • Bi C, Guosheng Pan Lu, Yang C-C, Hou M, Huang Y (2019b) Evacuation route recommendation using auto-encoder and Markov decision process. Appl Soft Comput 84(105741):1–11

    Google Scholar 

  • BöschenF, Scherp A (2015) Multi-oriented text extraction from information graphics. In: Proceedings of the 2015 ACM symposium on document engineering. ACM, pp 35–38

  • ChenZ, Cafarella M, Adar E (2015) Diagramflyer: a search engine for data-driven diagrams. In: Proceedings of the 24th international conference on World Wide Web. ACM, pp 183–186

  • Chen S, Li J, Andrienko G, Andrienko N, Wang Y et al (2020) Supporting story synthesis: bridging the gap between visual analytics and storytelling. IEEE Trans Vis Comput Graph 26(7):2499–2516

    Article  Google Scholar 

  • CholletF (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 1251–1258

  • ChoudhurySR, Wang S, Giles CL (2016) Scalable algorithms for scholarly figure mining and semantics. In: Proceedings of the international workshop on semantic big data. ACM, p 1

  • Cliche M, Rosenberg D, Madeka D, Yee C (2017a) Scatteract: automated extraction of data from scatter plots. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, Cham, pp 135–150

  • Cliché M, Rosenberg D, Madeka D et al (2017b) Scatteract: automated extraction of data from scatter plots. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, Cham, pp 135–150

  • Dai W, Wang M, Niu Z, Zhang J (2018) Chart decoder: generating textual and numeric information from chart images automatically. J Vis Lang Comput 48:101–109

    Article  Google Scholar 

  • DataThief. https://datathief.org/. Accessed 21 Oct 2019

  • Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  • HeK, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 770–778

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  • HuangW, Tan CL (2007) A system for understanding imaged infographics and its applications. In: Proceedings of the 2007 ACM symposium on document engineering. ACM, pp 9–18

  • Huang Z, Zhao Y, Chen W, Gao S, Yu K, Xu W, Tang M, Zhu M, Xu M (2020) A natural-language-based visual query approach of uncertain human trajectories. IEEE Trans Vis Comput Graph 26(1):1256–1266

    Google Scholar 

  • JayantC, Renzelmann M, Wen D, Krisnandi S, Ladner R, Comden D (2007) Automated tactile graphics translation: in the field. In: Proceedings of the 9th international ACM SIGACCESS conference on computers and accessibility. ACM, pp 75–82

  • JungD, Kim W, Song H, Hwang JI, Lee B, Kim B, Seo J (2017) ChartSense: interactive data extraction from chart images. In: Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, pp 6706–6717

  • Kong N, Agrawala M (2012) Graphical overlays: using layered elements to aid chart reading. IEEE Trans Vis Comput Graph 18(12):2631–2638

    Article  Google Scholar 

  • Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  • Lin TY, Maire M, Belongie S, Hays J, Perona P et al (2014) Microsoft coco: common objects in context. In: European conference on computer vision. Springer, Cham, pp 740–755

  • Liu L, Ouyang W, Wang X, Fieguth P, Chen J, Liu X, Pietikäinen M (2019a) Deep learning for generic object detection: a survey. Int J Comput Vis 128(1):261–318

    Google Scholar 

  • Liu X, Klabjan D, NBless P (2019) Data extraction from charts via single deep neural network. arXiv preprint arXiv:1906.11906

  • Ma Y, Tung AKH, Wang W, Gao X, Pan Z, Chen W (2020) ScatterNet: a deep subjective similarity model for visual analysis of scatterplots. IEEE Trans Vis Comput Graph 26(3):1562–1576

    Article  Google Scholar 

  • Matplotlib. https://matplotlib.org/. Accessed 21 Oct 2019

  • Mei H, Wei Y, Zhou S, Lin B, Zhao Y, Xia J, Chen W (2020a) RSATree: distribution-aware data representation of large-scale tabular datasets for flexible visual query. IEEE Trans Vis Comput Graph 26(1):1161–1171

    Article  Google Scholar 

  • Mei H, Guan H, Wen X, Chen W (2020b) DataV: data visualization on large high-resolution displays. Vis Inform 4(3):12–23

    Article  Google Scholar 

  • MéndezGG, Nacenta MA, Vandenheste S (2016) iVoLVER: interactive visual language for visualization extraction and reconstruction. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 4073–4085

  • Microsoft Project Oxford. https://www.projectoxford.ai/vision. Accessed 21 Oct 2019

  • Poco J, Heer J (2017) Reverse-engineering visualizations: recovering visual encodings from chart images. Comput Graph For 36(3):353–363

    Google Scholar 

  • Ren S, He K, Girshick R, Sun J (2017) Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  • SavvaM, Kong N, Chhajta A, Fei-Fei L, Agrawala M, Heer J (2011) Revision: automated classification, analysis and redesign of chart images. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM, pp 393–402

  • SiegelN, Horvitz Z, Levin R, Divvala S, Farhadi A (2016) FigureSeer: parsing result-figures in research papers. In: European conference on computer vision. Springer, Cham, pp 664–680

  • SmithR (2007) An overview of the Tesseract OCR engine. In: 9th international conference on document analysis and recognition. IEEE, pp 629–633

  • WebPlotDigitizer. https://automeris.io/WebPlotDigitizer/. Accessed 21 Oct 2019

  • Wei Y, Mei H, Zhao Y, Zhou S, Lin B, Jiang H, Chen W (2020) Evaluating perceptual bias during geometric scaling of scatterplots. IEEE Trans Vis Comput Graph 26(1):100–111

    Article  Google Scholar 

  • Xia J, Ye F, Chen W, Wang Y, Chen W, Ma Y, Tung AKH (2018) LDSScanner: exploratory analysis of low-dimensional structures in high-dimensional datasets. IEEE Trans Vis Comput Graph 24(1):236–245

    Article  Google Scholar 

  • XuK, Ba JL, Kiros R, Cho K, Courville A et al (2015) Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning. IEEE, pp 2048–2057

  • YuanJ, Chen C, Yang W, Liu M, Xia J, Liu S (2020) A survey of visual analytics techniques for machine learning. arXiv preprint arXiv:2008.09632

  • Zhao Y, Zhou F, Fan X, Liang X, Liu Y (2013) IDSRadar: a real-time visualization framework for IDS alerts. Sci China Inf Sci 56(8):1–12

    Article  Google Scholar 

  • Zhao Y, Wang L, Li S, Zhou F, Lin X, Lu Q, Ren L (2019) A visual analysis approach for understanding durability test data of automotive products. ACM Trans Intell Syst Technol 10(6):70–93

    Article  Google Scholar 

  • Zhao Y, Luo X, Lin X, Wang H, Kui X, Zhou F, Wang J, Chen Y, Chen W (2020) Visual analytics for electromagnetic situation awareness in radio monitoring and management. IEEE Trans Vis Comput Graph 26(1):590–600

    Article  Google Scholar 

  • Zhou ZH (2019) Abductive learning: towards bridging machine learning and logical reasoning. Sci China Inf Sci 62(7):191–193

    Article  MathSciNet  Google Scholar 

  • ZhouYP, Tan CL (2000) Hough technique for bar charts detection and recognition in document images. In: Proceedings 2000 international conference on image processing. IEEE, pp 605–608

  • Zhou F, Lin X, Liu C, Zhao Y, Xu P, Ren L, Xue T, Ren L (2019) A survey of visualization for smart manufacturing. J Vis 22(2):419–435

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science and Technology Fundamental Resources Investigation Program of China (No. 2018FY10090002), the National Natural Science Foundation of China (Nos. 61672538 and 61872388), and the Natural Science Foundation of Hunan Province (No. 2020JJ4758). The data sets and source codes of this work are available at Github: https://github.com/csuvis/BarchartReverseEngineering.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineer, Central South University, Changsha, China

    Fangfang Zhou, Yong Zhao, Wenjiang Chen, Yijing Tan, Yaqi Xu & Ying Zhao

  2. Beijing Key Laboratory of Big Data Technology for Food Safety, Beijing Technology and Business University, Beijing, China

    Yi Chen

  3. Institute of Systems Engineering, Academy of Military Sciences, Beijing, China

    Chao Liu

  4. Rail Data Research and Application Key Laboratory of Hunan Province, Changsha, China

    Ying Zhao

Authors
  1. Fangfang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenjiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yijing Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yaqi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ying Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Zhao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, F., Zhao, Y., Chen, W. et al. Reverse-engineering bar charts using neural networks. J Vis 24, 419–435 (2021). https://doi.org/10.1007/s12650-020-00702-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-020-00702-6

Keywords

Search

Navigation