Abstract
Having a comprehensive, high-quality dataset of road sign annotation is critical to the success of AI-based Road Sign Recognition (RSR) systems. In practice, annotators often face difficulties in learning road sign systems of different countries; hence, the tasks are often time-consuming and produce poor results. We propose a novel approach using knowledge graphs and a machine learning algorithm - variational prototyping-encoder (VPE) - to assist human annotators in classifying road signs effectively. Annotators can query the Road Sign Knowledge Graph using visual attributes and receive closest matching candidates suggested by the VPE model. The VPE model uses the candidates from the knowledge graph and a real sign image patch as inputs. We show that our knowledge graph approach can reduce sign search space by 98.9%. Furthermore, with VPE, our system can propose the correct single candidate for 75% of signs in the tested datasets, eliminating the human search effort entirely in those cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
These features are mandatory in all new cars sold within Europe from May 2022 [3].
- 2.
Note that we define “sign” as a prototypical sign, and differentiate it from “sign instance” which is an instance of a prototypical sign as seen in drive data. For example, if a set of images contains 4 stop signs and 4 yield signs, then there are 8 sign instances in total, and 2 signs (stop and yield).
- 3.
- 4.
References
Amazon Mechanical Turk. https://mturk.com. Accessed May 2020
ClickWorker. https://www.clickworker.com. Accessed May 2020
Regulation (EU) 2019/2144 of the European Parliament and of the Council. https://eur-lex.europa.eu/eli/reg/2019/2144/oj. Accessed May 2020
South African Road Traffic Signs Manual. https://www.transport.gov.za/. Accessed June 2020
UpWork. https://www.upwork.com/. Accessed May 2020
Belaroussi, R., Foucher, P., Tarel, J.P., Soheilian, B., Charbonnier, P., Paparoditis, N.: Road sign detection in images: a case study. In 20th International Conference on Pattern Recognition, pp. 484–488. IEEE (2010)
Bukhari, A.C., Nagy, M.L., Krauthammer, M., Ciccarese, P., Baker, C.J.: BIM: an open ontology for the annotation of biomedical images. In: ICBO (2015)
Chu, X., et al.: A data cleaning system powered by knowledge bases and crowdsourcing. In: Proceedings of the ACM SIGMOD International Conference on Management of Data (2015)
Inland Transport Committee: Convention on Road Traffic. Economic Commission for Europe, Vienna, Austria (1968)
Cowan, T.G.: JenaBean: easily bind JavaBeans to RDF. IBM DeveloperWorks (2008)
Dojchinovski, M., Reddy, D., Kliegr, T., Vitvar, T., Sack, H.: Crowdsourced corpus with entity salience annotations. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) (2016)
United Nations. Economic Commission for Europe. Transport Division. Convention on Road Traffic of 1968 and European Agreement Supplementing the Convention (2006 Consolidated Versions) (2007)
Hitzler, P., Krötzsch, M., Parsia, B., Patel-Schneider, P.F., Rudolph, S., et al.: OWL 2 web ontology language primer. W3C Recomm. 27(1), 123 (2009)
Jou, B., Chen, T., Pappas, N., Redi, M., Topkara, M., Chang, S.F.: Visual affect around the world: a large-scale multilingual visual sentiment ontology. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 59–168 (2015)
Kampffmeyer, M., Chen, Y., Liang, X., Wang, H., Zhang, Y., Xing, E.P.: Rethinking knowledge graph propagation for zero-shot learning. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
Kim, J., Oh, T.H., Lee, S., Pan, F., Kweon, I.S.: Variational prototyping-encoder: one-shot learning with prototypical images. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
Larsson F., Felsberg M.: Using Fourier descriptors and spatial models for traffic sign recognition. In: Scandinavian Conference on Image Analysis, pp. 238–249. Springer (2011)
Lopez, L.D., Fuentes, O.: Color-based road sign detection and tracking. In: Image Analysis and Recognition, pp. 1138–1147. Springer (2007)
Loy, G., Barnes, N.: Fast shape-based road sign detection for a driver assistance system. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 70–75. IEEE (2004)
Malik, R., Khurshid, J., Ahmad, S.N.: Road sign detection and recognition using colour segmentation, shape analysis and template matching. In: 2007 International Conference on Machine Learning and Cybernetics, vol. 6, pp. 3556–3560. IEEE (2007)
Mogelmose, A., Trivedi, M.M., Moeslund, T.B.: Vision-based traffic sign detection and analysis for intelligent driver assistance systems: perspectives and survey. IEEE Trans. Intell. Transp. Syst. 13(4), 1484–1497 (2012)
Musen, M.A.: The protégé project: a look back and a look forward. AI Matters 1(4), 4–12 (2015)
Texas MUTCD. Manual on uniform traffic control devices. Texas Department of Transportation, Austin (2006)
Acosta, M., Zaveri, A., Simperl, E., Kontokostas, D., Auer, S., Lehmann, J.: Crowdsourcing linked data quality assessment. In: International Semantic Web Conference (2013)
Stallkamp, J., Schlipsing, M., Salmen, J., Igel, C.: The German traffic sign recognition benchmark: a multi-class classification competition. In: International Joint Conference on Neural Networks (2011)
Timofte, R., Zimmermann, K., Van Gool, L.: Multi-view traffic sign detection, recognition, and 3D localisation. Mach. Vis. Appl. 25(3), 633–647 (2011). https://doi.org/10.1007/s00138-011-0391-3
Zhou, S., Deng, C., Piao, Z., Zhao, B.: Few-shot traffic sign recognition with clustering inductive bias and random neural network. Pattern Recogn. 100, 107160 (2019)
Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., Hu, S.: Traffic-sign detection and classification in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kim, J.E., Henson, C., Huang, K., Tran, T.A., Lin, WY. (2021). Accelerating Road Sign Ground Truth Construction with Knowledge Graph and Machine Learning. In: Arai, K. (eds) Intelligent Computing. Lecture Notes in Networks and Systems, vol 284. Springer, Cham. https://doi.org/10.1007/978-3-030-80126-7_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-80126-7_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-80125-0
Online ISBN: 978-3-030-80126-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)