Abstract
Mapping software is difficult to test because it is very costly to evaluate its output. This difficulty is generally known as the oracle problem, a fundamental challenge in software testing. In this paper, we propose a metamorphic testing strategy to alleviate the oracle problem in testing mapping software. We first conduct a case study to test Google Maps, the most popular web mapping service. The results of the case study show that our testing approach is effective, with the detection of several real-life bugs that can hardly be exposed under conventional testing paradigms. Following this, we conduct an analysis of the system OpenStreetMap, well-known open-source mapping software built and maintained by a community of users. We show the potential of metamorphic testing for such systems. These case studies show that metamorphic testing can be applied to mapping software for both verification and validation purposes.
An initial version of this paper was presented at HICSS-51 [1].
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brown, J., Zhou, Z.Q., Chow, Y.-W.: Metamorphic testing of navigation software: a pilot study with Google Maps. In: Proceedings of the 51st Annual Hawaii International Conference on System Sciences (HICSS-51), pp. 5687–5696 (2018). http://hdl.handle.net/10125/50602
The world factbook: Central Intelligence Agency (2013). https://www.cia.gov/library/publications/the-world-factbook/fields/2085.html
BuiltWith: Mapping usage statistics (2017). https://trends.builtwith.com/mapping
OpenStreetMap stats report: OpenStreetMap Foundation (2018). https://www.openstreetmap.org/stats/data_stats.html
Stats: OpenStreetMap Foundation (2018). https://wiki.openstreetmap.org/wiki/Stats
List of OSM-based services: OpenStreetMap Foundation (2018). https://wiki.openstreetmap.org/wiki/List_of_OSM-based_services
OSM Internet Links: OpenStreetMap Foundation (2018). https://wiki.openstreetmap.org/wiki/OSM_Internet_Links
Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S.: The oracle problem in software testing: a survey. IEEE Trans. Softw. Eng. 41(5), 507–525 (2015)
Wright, M., Stallings, D., Dunn, D.: The effectiveness of global positioning system electronic navigation. In: Proceedings of IEEE SoutheastCon, pp. 62–67 (2003)
Chen, T.Y., Tse, T.H., Zhou, Z.Q.: Fault-based testing without the need of oracles. Inf. Softw. Technol. 45(1), 1–9 (2003)
Chen, T.Y., Kuo, F.-C., Zhou, Z.Q.: An effective testing method for end-user programmers. ACM SIGSOFT Softw. Eng. Notes 30(4), 1–5 (2005)
Liu, H., Kuo, F.-C., Towey, D., Chen, T.Y.: How effectively does metamorphic testing alleviate the oracle problem? IEEE Trans. Softw. Eng. 40(1), 4–22 (2014)
Lindvall, M., Ganesan, D., Árdal, R., Wiegand, R.E.: Metamorphic model-based testing applied on NASA DAT – an experience report. In: Proceedings of the 37th International Conference on Software Engineering (ICSE 2015), pp. 129–138 (2015)
Zhou, Z.Q., Xiang, S., Chen, T.Y.: Metamorphic testing for software quality assessment: a study of search engines. IEEE Trans. Softw. Eng. 42(3), 264–284 (2016)
Segura, S., Fraser, G., Sanchez, A.B., Ruiz-Cortés, A.: A survey on metamorphic testing. IEEE Trans. Softw. Eng. 42(9), 805–824 (2016)
Chen, T.Y., et al.: Metamorphic testing for cybersecurity. Computer 49(6), 48–55 (2016)
Kanewala, U., Pullum, L.L., Segura, S., Towey, D., Zhou, Z.Q.: Message from the workshop chairs. In: Proceedings of the IEEE/ACM 1st International Workshop on Metamorphic Testing (ICSE MET 2016), in Conjunction with the 38th International Conference on Software Engineering (ICSE). ACM Press (2016)
Jarman, D.C., Zhou, Z.Q., Chen, T.Y.: Metamorphic testing for Adobe data analytics software. In: Proceedings of the IEEE/ACM 2nd International Workshop on Metamorphic Testing (ICSE MET 2017), in Conjunction with the 39th International Conference on Software Engineering (ICSE), pp. 21–27 (2017)
Ding, J., Hu, X.-H., Gudivada, V.: A machine learning based framework for verification and validation of massive scale image data. IEEE Trans. Big Data. https://doi.org/10.1109/TBDATA.2017.2680460
Pezzè, M., Young, M.: Software Testing and Analysis: Process, Principles, and Techniques. Wiley, New York (2008)
Chen, T.Y., Huang, D.H., Tse, T.H., Zhou, Z.Q.: Case studies on the selection of useful relations in metamorphic testing. In: Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004). Polytechnic University of Madrid, pp. 569–583 (2004)
Goodman, J.L.: The space shuttle and GPS: a safety-critical navigation upgrade. In: Erdogmus, H., Weng, T. (eds.) ICCBSS 2003. LNCS, vol. 2580, pp. 92–100. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36465-X_9
Nisner, P.D., Johannessen, R.: Ten million data points from TSO-approved aviation GPS receivers: results of analysis and applications to design and use in aviation. Navigation 47(1), 43–50 (2000)
Elleuch, W., Wali, A., Alimi, A.M.: Mining road map from big database of GPS data. In: 14th International Conference on Hybrid Intelligent Systems (HIS), pp. 193–198. IEEE (2014)
Elleuch, W., Wali, A., Alimi, A.M.: Collection and exploration of GPS based vehicle traces database. In: 4th International Conference on Advanced Logistics and Transport (ICALT), pp. 275–280. IEEE (2015)
Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: a new approach for generating next test cases, Technical report HKUST-CS98-01, Department of Computer Science, Hong Kong Univ. of Science and Technology (1998)
Goodman, J.L.: A software perspective on GNSS receiver integration and operation. In: Rycroft, M. (ed.) Satellite Navigation Systems. Space Studies, vol. 8, pp. 119–126. Springer, Dordrecht (2003). https://doi.org/10.1007/978-94-017-0401-4_13
Luxen, D., Vetter, C.: Real-time routing with OpenStreetMap data. In: Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (GIS 2011), pp. 513–516. ACM (2011)
Graf, F., Kriegel, H.-P., Renz, M., Schubert, M.: MARiO: multi-attribute routing in open street map. In: Pfoser, D. (ed.) SSTD 2011. LNCS, vol. 6849, pp. 486–490. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22922-0_36
Cipeluch, B., Jacob, R., Winstanley, A., Mooney, P.: Comparison of the accuracy of OpenStreetMap for Ireland with Google Maps and Bing Maps. In: Proceedings of the 9th International Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Sciences, pp. 337–341 (2010)
Chen, T.Y., et al.: Metamorphic testing: a review of challenges and opportunities. ACM Comput. Surv. 51(1), 4:1–4:27 (2018)
Zhou, Z.Q., Sun, L., Chen, T.Y., Towey, D.: Metamorphic relations for enhancing system understanding and use. IEEE Trans. Softw. Eng. https://doi.org/10.1109/TSE.2018.2876433
Zhou, Z.Q., Sun, L.: Metamorphic testing of driverless cars. Commun. ACM 62(3), 61–67 (2019)
Segura, S., Parejo, J.A., Troya, J., Ruiz-Cortés, A.: Metamorphic testing of RESTful Web APIs. IEEE Trans. Softw. Eng. 44(11), 1083–1099 (2018)
Google Maps Directions API: Google (2016). https://developers.google.com/maps/documentation/directions/intro
Acknowledgements
This work was supported in part by a linkage grant of the Australian Research Council (project ID: LP160101691) and an Australian Government Research Training Program scholarship.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Brown, J., Zhou, Z.Q., Chow, YW. (2019). Metamorphic Testing of Mapping Software. In: Majchrzak, T., Mateos, C., Poggi, F., Grønli, TM. (eds) Towards Integrated Web, Mobile, and IoT Technology. Lecture Notes in Business Information Processing, vol 347. Springer, Cham. https://doi.org/10.1007/978-3-030-28430-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-28430-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28429-9
Online ISBN: 978-3-030-28430-5
eBook Packages: Computer ScienceComputer Science (R0)