Skip to main content

Accessible Chemical Structural Formulas Through Interactive Document Labeling

  • Conference paper
  • First Online:
Computers Helping People with Special Needs (ICCHP-AAATE 2022)


Despite a number of advances in the accessibility of STEM education, there is a lack of advanced tool support for authors and educators seeking to make corresponding documents accessible. We propose an interactive labeling method that combines an AI with user input to create accessible chemical structural formulas and incrementally improve the model. The model is a deep learning method based on a convolutional neural network and a transformer-based encoder-decoder. We implement this in a tool that enables graphical labeling of structural formulas and supports the user by performing a similarity search to suggest matches. Our approach aims to improve both the efficiency and effectiveness of labeling chemical structural formulas for accessibility purposes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. 1.


  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  2. Deng, Y., Kanervisto, A., Rush, A.M.: What you get is what you see: A visual markup decompiler. arXiv preprint arXiv:1609.04938 (2016)

  3. Heller, S.R., McNaught, A., Pletnev, I., Stein, S., Tchekhovskoi, D.: InChI, the IUPAC international chemical identifier. J. Cheminformatics 7(1), 1–34 (2015)

    Google Scholar 

  4. Jiang, C., Jin, X., Dong, Y., Chen, M.: Kekule.js: an open source javascript chemoinformatics toolkit. J. Chem. Inf. Mod. 56(6), 1132–1138 (2016)

    Google Scholar 

  5. Khokhlov, I., Krasnov, L., Fedorov, M.V., Sosnin, S.: Image2smiles: transformer-based molecular optical recognition engine. Chem.-Meth. 2(1), e202100069 (2022)

    Google Scholar 

  6. Kim, S., Thiessen, P., Cheng, T., Yu, B., Bolton, E.: An update on PUG-REST: RESTful interface for programmatic access to PubChem. Nucleic Acids Res. 46(W1), W563–W570 (2018)

    Google Scholar 

  7. McGrath, M., Brown, J.: Visual learning for science and engineering. IEEE Comput. Graph. Appl. 25, 56–63 (2005)

    Google Scholar 

  8. Nadj, M., Knaeble, M., Li, M.X., Maedche, A.: Power to the oracle? design principles for interactive labeling systems in machine learning. KI - Künstliche Intelligenz 34(2), 131–142 (2020).

  9. Park, J., Rosania, G.R., Shedden, K.A., Nguyen, M., Lyu, N., Saitou, K.: Automated extraction of chemical structure information from digital raster images. Chem. Central J. 6 (2009)

    Google Scholar 

  10. Rajan, K., Zielesny, A., Steinbeck, C.: DECIMER 1.0: deep learning for chemical image recognition using transformers. J. Cheminformatics 13(1), 61 (2021)

    Google Scholar 

  11. Sadawi, N.M., Sexton, A.P., Sorge, V.: Chemical structure recognition: a rule-based approach. In: Document Recognition and Retrieval XIX, vol. 8297, pp. 101–109. SPIE (2012)

    Google Scholar 

  12. Schwarz, T., Rajgopal, S., Stiefelhagen, R.: Accessible EPUB: making EPUB 3 documents universal accessible. In: Miesenberger, K., Kouroupetroglou, G. (eds.) ICCHP 2018. LNCS, vol. 10896, pp. 85–92. Springer, Cham (2018).

  13. Shave, S., Auer, M.: SimilarityLab: molecular similarity for SAR exploration and target prediction on the web. Processes 9(9) (2021)

    Google Scholar 

  14. Sorge, V.: Polyfilling accessible chemistry diagrams. In: Miesenberger, K., Bühler, C., Penaz, P. (eds.) ICCHP 2016. LNCS, vol. 9758, pp. 43–50. Springer, Cham (2016).

  15. Staker, J., Marshall, K., Abel, R., McQuaw, C.M.: Molecular structure extraction from documents using deep learning. J. Chem. Inf. Mod. 59(3), 1017–1029 (2019)

    Google Scholar 

  16. Valko, A.T., Johnson, A.P.: CLiDE Pro: the latest generation of CLiDE, a tool for optical chemical structure recognition. J. Chem. Inf. Model. 49(4), 780–787 (2009)

    Google Scholar 

  17. Vaswani, A., et al.: Attention is all you need. In: NeurIPS, vol. 30. Curran Associates, Inc. (2017)

    Google Scholar 

  18. in’t Veld, D., Sorge, V.: The Dutch Best Practice for Teaching Chemistry Diagrams to the Visually Impaired. In: Miesenberger, K., Kouroupetroglou, G. (eds.) ICCHP 2018. LNCS, vol. 10896, pp. 644–647. Springer, Cham (2018).

    Chapter  Google Scholar 

  19. Weininger, D.: SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules. J. Chem. Inf. Comput. Sci. 28(1), 31–36 (1988)

    Google Scholar 

  20. Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning (ICML), pp. 2048–2057. PMLR (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Merlin Knaeble .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Knaeble, M. et al. (2022). Accessible Chemical Structural Formulas Through Interactive Document Labeling. In: Miesenberger, K., Kouroupetroglou, G., Mavrou, K., Manduchi, R., Covarrubias Rodriguez, M., Penáz, P. (eds) Computers Helping People with Special Needs. ICCHP-AAATE 2022. Lecture Notes in Computer Science, vol 13341. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-08647-2

  • Online ISBN: 978-3-031-08648-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics