Abstract
Illustrations are widely used in education, and sometimes, alternatives are not available for visually impaired students. Therefore, those students would benefit greatly from an automatic illustration description system, but only if those descriptions were complete, correct, and easily understandable using a screenreader. In this paper, we report on a study for the assessment of automated image descriptions. We interviewed experts to establish evaluation criteria, which we then used to create an evaluation questionnaire for sighted non-expert raters, and description templates. We used this questionnaire to evaluate the quality of descriptions which could be generated with a template-based automatic image describer. We present evidence that these templates have the potential to generate useful descriptions, and that the questionnaire identifies problems with description templates.
This work is financially supported by the German Federal Ministry of Education and Research (BMBF) and the European Social Fund (ESF) (Project InclusiveOCW, no. 01PE17004).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ferres, L., Parush, A., Roberts, S., Lindgaard, G.: Helping people with visual impairments gain access to graphical information through natural language: the iGraph system. In: Miesenberger, K., Klaus, J., Zagler, W.L., Karshmer, A.I. (eds.) ICCHP 2006. LNCS, vol. 4061, pp. 1122–1130. Springer, Heidelberg (2006). https://doi.org/10.1007/11788713_163
Jessen, M., Böschen, F., Scherp, A.: Text localization in scientific figures using fully convolutional neural networks on limited training data. In: Schimmler, S., Borghoff, U.M. (eds.) Proceedings of the CM Symposium on Document Engineering 2019, Berlin, Germany, 23–26 September 2019, pp. 13:1–13:10. ACM (2019). https://doi.org/10.1145/3342558.3345396
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A diagram is worth a dozen images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 235–251. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_15
Le, A.D., Indurkhya, B., Nakagawa, M.: Pattern generation strategies for improving recognition of handwritten mathematical expressions. Pattern Recogn. Lett. 128, 255–262 (2019). https://doi.org/10.1016/j.patrec.2019.09.002
Liu, L., Ouyang, W., Wang, X., Fieguth, P.W., Chen, J., Liu, X., Pietikäinen, M.: Deep learning for generic object detection: a survey. Int. J. Comput. Vis. 128(2), 261–318 (2020). https://doi.org/10.1007/s11263-019-01247-4
Morash, V.S., Siu, Y., Miele, J.A., Hasty, L., Landau, S.: Guiding novice web workers in making image descriptions using templates. TACCESS 7(4), 12:1–12:21 (2015). https://doi.org/10.1145/2764916
Morris, D., Tang, P., Ewerth, R.: A neural approach for text extraction from scholarly figures. In: 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, 20–25 September 2019, pp. 1438–1443. IEEE (2019). https://doi.org/10.1109/ICDAR.2019.00231
Morris, M.R., Johnson, J., Bennett, C.L., Cutrell, E.: Rich representations of visual content for screen reader users. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, Montreal, QC, Canada, 21–26 April 2018. p. 59 (2018). https://doi.org/10.1145/3173574.3173633
NCAM, DIAGRAM: Image description guidelines. http://diagramcenter.org/table-of-contents-2.html
Park, C.C., Kim, B., Kim, G.: Attend to you: Personalized image captioning with context sequence memory networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 6432–6440. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.681
Reid, L.G., Snow-Weaver, A.: WCAG 2.0: a web accessibility standard for the evolving web. In: Yesilada, Y., Sloan, D. (eds.) Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A 2008, Beijing, China, 21–22 April 2008, pp. 109–115. ACM International Conference Proceeding Series, ACM (2008).https://doi.org/10.1145/1368044.1368069
Shuster, K., Humeau, S., Hu, H., Bordes, A., Weston, J.: Engaging image captioning via personality. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 12516–12526. Computer Vision Foundation / IEEE (2019). https://doi.org/10.1109/CVPR.2019.01280
Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 24th International Conference on Pattern Recognition, ICPR 2018, Beijing, China, 20–24 August 2018, pp. 2245–2250. IEEE Computer Society (2018). https://doi.org/10.1109/ICPR.2018.8546031
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Hoppe, A., Morris, D., Ewerth, R. (2021). Evaluation of Automated Image Descriptions for Visually Impaired Students. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2021. Lecture Notes in Computer Science(), vol 12749. Springer, Cham. https://doi.org/10.1007/978-3-030-78270-2_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-78270-2_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78269-6
Online ISBN: 978-3-030-78270-2
eBook Packages: Computer ScienceComputer Science (R0)