Abstract
The development of caption metrics is relatively new in the accessibility research community. However, little work has been done comparing the effectiveness of newly developed caption metrics. More specifically, in low accuracy settings such as live television, where users report the most difficulty using captions. Through a user study with fifteen participants, we compared two caption metrics systems, Word Error Rate (WER) and Automated-Caption Evaluation (ACE), for their accuracy in evaluating caption quality in live television. We compared human-perceived quality statistics with each caption metric’s data. Analysis of the correlation between human statistics and each caption metric found that WER had a slightly higher correlation with participants. We found that ACE was more sensitive to errors that WER, and penalized captions more than participants. However, the difference in performance between WER and ACE was not statistically significant, and neither WER nor ACE are optimized for use with live television captioning. Future work should explore how caption metrics could be better optimized for use with live television.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jang, P.J., Hauptmann, A.G.: Improving acoustic models with captioned multimedia speech. In: Proceedings IEEE International Conference on Multimedia Computing and Systems, vol. 2, pp. 767–771. IEEE, June 1999
Block, M.H., Okrand, M.: Real-time closed-captioned television as an educational tool. Am. Ann. Deaf 128(5), 636–641 (1983)
Apone, T., Brooks, M., O’Connell, T.: Caption Accuracy Metrics Project. Caption Viewer Survey: Error Ranking of Real-time Captions in Live Television News Programs. Boston (2010)
Al Amin, A.: Audio-Visual Caption Evaluation Metric for People who are Deaf and Hard of Hearing (2020)
Apone, T., Botkin, B., Brooks, M., Goldberg, L.: Research into Automated Error Ranking of Real-time Captions in Live Television News Programs. The Carl and Ruth Shapiro Family National Center for Accessible Media at WGBH (NCAM) (2011)
Kafle, S., Huenerfauth, M.: Evaluating the usability of automatically generated captions for people who are deaf or hard of hearing. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 165–174, October 2017
Acknowledgements
The contents of this paper were developed in part under a grant from the National Science Foundation, grant #1757836 (REU AICT) and under a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number #90DPCP0002). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). The contents of this poster do not necessarily represent the policy of NIDILRR, ACL, HHS, and you should not assume endorsement by the Federal Government. We thank Akhter Al-Amin and Mariana Arroyo Chavez for their help with calculating the caption metrics for the videos.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Wells, T., Christoffels, D., Vogler, C., Kushalnagar, R. (2022). Comparing the Accuracy of ACE and WER Caption Metrics When Applied to Live Television Captioning. In: Miesenberger, K., Kouroupetroglou, G., Mavrou, K., Manduchi, R., Covarrubias Rodriguez, M., Penáz, P. (eds) Computers Helping People with Special Needs. ICCHP-AAATE 2022. Lecture Notes in Computer Science, vol 13341. Springer, Cham. https://doi.org/10.1007/978-3-031-08648-9_61
Download citation
DOI: https://doi.org/10.1007/978-3-031-08648-9_61
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08647-2
Online ISBN: 978-3-031-08648-9
eBook Packages: Computer ScienceComputer Science (R0)