Advertisement

(A)kNN Query Processing on the Cloud: A Survey

  • Nikolaos NodarakisEmail author
  • Angeliki Rapti
  • Spyros Sioutas
  • Athanasios K. Tsakalidis
  • Dimitrios Tsolis
  • Giannis Tzimas
  • Yannis Panagis
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10230)

Abstract

A k-nearest neighbor (kNN) query determines the k nearest points, using distance metrics, from a given location. An all k-nearest neighbor (AkNN) query constitutes a variation of a kNN query and retrieves the k nearest points for each point inside a database. Their main usage resonates in spatial databases and they consist the backbone of many location-based applications and not only. Although (A)kNN is a fundamental query type, it is computationally very expensive. During the last years a multiplicity of research papers has focused around the distributed (A)kNN query processing on the cloud. This work constitutes a survey of research efforts towards this direction. The main contribution of this work is an up-to-date review of the latest (A)kNN query processing approaches. Finally, we discuss various research challenges and directions of further research around this domain.

Keywords

Big data Nearest neighbor MapReduce NoSQL Query processing 

References

  1. 1.
    Abbasifard, M.R., Ghahremani, B., Naderi, H.: A survey on nearest neighbor search methods. Int. J. Comput. Appl. 95, 39–52 (2014)Google Scholar
  2. 2.
    Abdelsadek, A., Hefeeda, M.: DIMO: distributed index for matching multimedia objects using MapReduce. In: Proceedings of the 5th ACM Multimedia Systems Conference, pp. 115–126. ACM, New York (2014)Google Scholar
  3. 3.
    Aji, A., Wang, F.: High performance spatial query processing for large scale scientific data. In: Proceedings of the on SIGMOD/PODS 2012 PhD Symposium, pp. 9–14. ACM, New York (2012)Google Scholar
  4. 4.
    Aji, A., Wang, F., Saltz, J.H.: Towards building a high performance spatial query system for large scale medical imaging data. In: Proceedings of the 20th International Conference on Advances in Geographic Information Systems, pp. 309–318. ACM, New York (2012)Google Scholar
  5. 5.
    Aji, A., Wang, F., Vo, H., Lee, R., Liu, Q., Zhang, X., Saltz, J.: Hadoop GIS: a high performance spatial data warehousing system over MapReduce. Proc. VLDB Endow. 6, 1009–1020 (2013)CrossRefGoogle Scholar
  6. 6.
    Akdogan, A., Demiryurek, U., Kashani, F.B., Shahabi, C.: Voronoi-based geospatial query processing with MapReduce. In: Proceedings of the IEEE 2nd International Conference on Cloud Computing Technology and Science, pp. 9–16. IEEE Computer Society, Washington, DC (2010)Google Scholar
  7. 7.
    Aly, M., Munich, M., Perona, P.: Distributed Kd-trees for retrieval from very large image collections. In: Proceedings of the British Machine Vision Conference (BMVC) (2011)Google Scholar
  8. 8.
    Andreica, M.I., Tapus, N.: Sequential and MapReduce-based algorithms for constructing an in-place multidimensional quad-tree index for answering fixed-radius nearest neighbor queries. Acta Universitatis Apulensis - Mathematics-Informatics, pp. 131–151 (2012)Google Scholar
  9. 9.
    Baig, F., Mehrotra, M., Vo, H., Wang, F., Saltz, J., Kurc, T.: SparkGIS: efficient comparison and evaluation of algorithm results in tissue image analysis studies. In: Wang, F., Luo, G., Weng, C., Khan, A., Mitra, P., Yu, C. (eds.) Big-O(Q)/DMAH -2015. LNCS, vol. 9579, pp. 134–146. Springer, Cham (2016). doi: 10.1007/978-3-319-41576-5_10 Google Scholar
  10. 10.
    Bhatia, N.: Vandana: Survey of Nearest Neighbor Techniques. CoRR abs/1007.0085 (2010)Google Scholar
  11. 11.
    Böhm, C., Krebs, F.: The k-nearest neighbour join: turbo charging the KDD process. Knowl. Inf. Syst. 6, 728–749 (2004)CrossRefGoogle Scholar
  12. 12.
    Candan, K.S., Nagarkar, P., Nagendra, M., Yu, R.: RanKloud: a scalable ranked query processing framework on hadoop. In: Proceedings of the 14th International Conference on Extending Database Technology, pp. 574–577. ACM, New York (2011)Google Scholar
  13. 13.
    Cary, A., Sun, Z., Hristidis, V., Rishe, N.: Experiences on processing spatial data with MapReduce. In: Winslett, M. (ed.) SSDBM 2009. LNCS, vol. 5566, pp. 302–319. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02279-1_24 CrossRefGoogle Scholar
  14. 14.
    Cech, P., Kohout, J., Lokoc, J., Komárek, T., Marousek, J., Pevný, T.: Feature extraction and malware detection on large HTTPS data using MapReduce. In: Amsaleg, L., Houle, M.E., Schubert, E. (eds.) SISAP 2016. LNCS, vol. 9939, pp. 311–324. Springer, Cham (2016). doi: 10.1007/978-3-319-46759-7_24
  15. 15.
    Chatzimilioudis, G., Costa, C., Zeinalipour-Yazti, D., Lee, W.-C., Pitoura, E.: Distributed in-memory processing of all k nearest neighbor queries. IEEE Trans. Knowl. Data Eng. 28, 925–938 (2016)CrossRefGoogle Scholar
  16. 16.
    Chen, Y., Patel, J.M.: Efficient evaluation of all-nearest-neighbor queries. In: Proceedings of the 23rd IEEE International Conference on Data Engineering, pp. 1056–1065. IEEE Computer Society, Washington, DC (2007)Google Scholar
  17. 17.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of the 6th Symposium on Operating Systems Design and Implementation, pp. 137–150. USENIX Association, Berkeley (2004)Google Scholar
  18. 18.
    Deng, Z., Zhu, X., Cheng, D., Zong, M., Zhang, S.: Efficient kNN classification algorithm for big data. Neurocomputing 195, 143–148 (2016)CrossRefGoogle Scholar
  19. 19.
    Dhanabal, S., Chandramathi, S.: A review of various k-nearest neighbor query processing techniques. Int. J. Comput. Appl. 31, 14–22 (2011)Google Scholar
  20. 20.
    Dong, X., Feifei, L., Bin, Y., Gefei, L., Liang, Z., Minyi, G.: Simba: efficient in-memory spatial analytics. In: Proceedings of the 2016 International Conference on Management of Data, pp. 1071–1085. ACM, New York (2016)Google Scholar
  21. 21.
    Du, Q., Li, X.: A novel KNN join algorithms based on Hilbert R-tree in MapReduce. In: Proceedings of the 3rd International Conference on Computer Science and Network Technology, pp. 417–420. IEEE (2013)Google Scholar
  22. 22.
    Eldawy, A., Mokbel, M.F.: SpatialHadoop: a MapReduce framework for spatial data. In: Proceedings of the 31st IEEE International Conference on Data Engineering, pp. 1352–1363. IEEE Computer Society, Washington, DC (2015)Google Scholar
  23. 23.
    Emrich, T., Graf, F., Kriegel, H.-P., Schubert, M., Thoma, M.: Optimizing all-nearest-neighbor queries with trigonometric pruning. In: Gertz, M., Ludäscher, B. (eds.) SSDBM 2010. LNCS, vol. 6187, pp. 501–518. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-13818-8_35 CrossRefGoogle Scholar
  24. 24.
    Gkoulalas-Divanis, A., Verykios, V.S., Bozanis, P.: A network aware privacy model for online requests in trajectory data. Data Knowl. Eng. 68, 431–452 (2009)CrossRefGoogle Scholar
  25. 25.
    Ioup, E., Shaw, K., Sample, J., Abdelguerfi, M.: Efficient AKNN spatial network queries using the M-Tree. In: Proceedings of the 15th Annual ACM International Symposium on Advances in Geographic Information Systems, pp. 46:1–46:4. ACM, New York (2007)Google Scholar
  26. 26.
    Ji, C., Dong, T., Li, Y., Shen, Y., Li, K., Qiu, W., Qu, W., Guo, M.: Inverted grid-based kNN query processing with MapReduce. In: Proceedings of the 7th ChinaGrid Annual Conference, pp. 25–32 (2012)Google Scholar
  27. 27.
    Ji, C., Li, Z., Qu, W., Xu, Y., Li, Y.: Scalable nearest neighbor query processing based on Inverted Grid Index. J. Network Comput. Appl. 44, 172–182 (2014)CrossRefGoogle Scholar
  28. 28.
    Kokotinis, I., Kendea, M., Nodarakis, N., Rapti, A., Sioutas, S., Tsakalidis, A.K., Tsolis, D., Panagis, Y.: NSM-Tree: efficient indexing on top of NoSQL databases. In: Post-proceedings of the 2nd International Workshop on Algorithmic Aspects of Cloud Computing (2016)Google Scholar
  29. 29.
    Liao, H., Jizhong, H., Jinyun, F.: Multi-dimensional index on hadoop distributed file system. In: Proceedings of the 2010 IEEE Fifth International Conference on Networking, Architecture, and Storage, pp. 240–249. IEEE Computer Society, Washington, DC, USA (2010)Google Scholar
  30. 30.
    Liu, T., Rosenberg, C., Rowley, H.A.: Clustering billions of images with large scale nearest neighbor search. In: Proceedings of the 8th IEEE Workshop on Applications of Computer Vision, p. 28. IEEE Computer Society (2007)Google Scholar
  31. 31.
    Lu, P., Chen, G., Ooi, B.C., Vo, H.T., Wu, S.: ScalaGiST: scalable generalized search trees for mapreduce systems [Innovative Systems Paper]. Proc. VLDB Endow. 7, 1797–1808 (2014)CrossRefGoogle Scholar
  32. 32.
    Lu, W., Shen, Y., Chen, S., Ooi, B.C.: Efficient processing of k nearest neighbor joins using mapreduce. Proc. VLDB Endow. 5, 1016–1027 (2012)CrossRefGoogle Scholar
  33. 33.
    Mahapatra, R.P., Chakraborty, P.S.: Comparative analysis of nearest neighbor query processing techniques. Procedia Comput. Sci. 57, 1289–1298 (2015)CrossRefGoogle Scholar
  34. 34.
    Maillo, J., Ramireza, S., Triguero, I., Herrera, F.: kNN-IS: an iterative spark-based design of the k-nearest neighbors classifier for big data. Knowledge-Based Systems (2016, in press)Google Scholar
  35. 35.
    Maillo, J., Triguero, I., Herrera, F.: A MapReduce-based k-nearest neighbor approach for big data classification. In: IEEE TrustCom/BigDataSE/ISPA, pp. 167–172. IEEE Computer Society, Washington, DC (2015)Google Scholar
  36. 36.
    Maleki, E.F., Azadani, M.N., Ghadiri, N.: Performance evaluation of spatialhadoop for big web mapping data. In: Proceedings of the 2016 Second International Conference on Web Research. IEEE Computer Society, Washington, DC (2016, to be published)Google Scholar
  37. 37.
    Muja, M., Lowe, D.G.: Scalable nearest neighbor algorithms for high dimensional data. IEEE Trans. Pattern Anal. Mach. Intell. 36, 2227–2240 (2014)CrossRefGoogle Scholar
  38. 38.
    Naami, K.M.A., Seker, S., Khan, L.: GISQF: an efficient spatial query processing system. In: Proceedings of the 2014 IEEE International Conference on Cloud Computing, pp. 681–688. IEEE Computer Society, Washington, DC (2014)Google Scholar
  39. 39.
    Nishimura, S., Das, S., Agrawal, D., Abbadi, A.E.: MD-HBase: a scalable multi-dimensional data infrastructure for location aware services. In: Proceedings of the 2011 IEEE 12th International Conference on Mobile Data Management, vol. 01, pp. 7–16. IEEE Computer Society, Washington, DC (2011)Google Scholar
  40. 40.
    Nodarakis, N., Pitoura, E., Sioutas, S., Tsakalidis, A., Tsoumakos, D., Tzimas, G.: Efficient multidimensional AkNN query processing in the cloud. In: Decker, H., Lhotská, L., Link, S., Spies, M., Wagner, R.R. (eds.) DEXA 2014. LNCS, vol. 8644, pp. 477–491. Springer, Cham (2014). doi: 10.1007/978-3-319-10073-9_41 Google Scholar
  41. 41.
    Nodarakis, N., Pitoura, E., Sioutas, S., Tsakalidis, A., Tsoumakos, D., Tzimas, G.: kdANN+: a rapid AkNN classifier for big data. Trans. Large-Scale Data Knowl. Centered Syst. 24, 139–168 (2016)Google Scholar
  42. 42.
    Nodarakis, N., Sioutas, S., Tsakalidis, A., Tzimas, G.: Large scale sentiment analysis on Twitter with spark. In: Proceedings of the Workshops of the EDBT/ICDT 2016 Joint Conference, CEUR Workshop Proceedings, vol. 1558 (2016). CEUR-WS.org
  43. 43.
    Nodarakis, N., Sioutas, S., Tsakalidis, A., Tzimas, G.: MR-SAT: a MapReduce algorithm for big data sentiment analysis on Twitter. In: Proceedings of the 12th International Conference on Web Information Systems and Technologies, vol. 1, pp. 140–147. SciTePress (2016)Google Scholar
  44. 44.
    Plaku, E., Kavraki, L.E.: Distributed computation of the knn graph for large high-dimensional point sets. J. Parallel Distrib. Comput. 67, 346–359 (2007)CrossRefzbMATHGoogle Scholar
  45. 45.
    Reyes-Ortiz, J.L., Oneto, L., Anguita, D.: Big data analytics in the cloud: spark on Hadoop vs MPI/OpenMP on Beowulf. Procedia Comput. Sci. 53, 121–130 (2015)CrossRefGoogle Scholar
  46. 46.
    Roussopoulos, N., Kelley, S., Vincent, F.: Nearest neighbor queries. In: Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, pp. 71–79. ACM, New York (1995)Google Scholar
  47. 47.
    Song, G., Rochas, J., Huet, F., Magoulès, F.: Solutions for processing K nearest neighbor joins for massive data on MapReduce. In: Proceedings of the 23rd Euromicro International Conference on Parallel, Distributed and Network-based Processing, March 2015, Turku, Finland (2015)Google Scholar
  48. 48.
    Song, G., Rochas, J., Huet, F., Magoulès, F.: K nearest neighbour joins for big data on MapReduce: a theoretical and experimental analysis. IEEE Trans. Knowl. Data Eng. 28, 2376–2392 (2016)CrossRefGoogle Scholar
  49. 49.
    Stupar, A., Michel, S., Schenkel, R.: RankReduce - processing K-nearest neighbor queries on top of MapReduce. In: Proceedings of the 8th Workshop on Large-Scale Distributed Systems for Information Retrieval, pp. 13–18. ACM, New York (2010)Google Scholar
  50. 50.
    Sun, K., Kang, H., Park, H.-H.: Tagging and classifying facial images in cloud environments based on KNN using MapReduce. Optik - Int. J. Light Electron Optics 126, 3227–3233 (2015)CrossRefGoogle Scholar
  51. 51.
    Sun, Z., Zhang, H., Liu, Z., Xu, C., Wang, L.: Migrating GIS big data computing from Hadoop to Spark: an exemplary study Using Twitter. In: Proceedings of the IEEE 9th International Conference on Cloud Computing, pp. 351–358. IEEE Computer Society, Washington, DC (2016)Google Scholar
  52. 52.
    Talavera-Llames, R.L., Pérez-Chacón, R., Martínez-Ballesteros, M., Troncoso, A., Martínez-Álvarez, F.: A nearest neighbours-based algorithm for big time series data forecasting. In: Martínez-Álvarez, F., Troncoso, A., Quintián, H., Corchado, E. (eds.) HAIS 2016. LNCS (LNAI), vol. 9648, pp. 174–185. Springer, Cham (2016). doi: 10.1007/978-3-319-32034-2_15 CrossRefGoogle Scholar
  53. 53.
    Tang, M., Yu, Y., Malluhi, Q.M., Ouzzani, M., Aref, W.G.: LocationSpark: a distributed in-memory data management system for big spatial data. PVLDB 9, 1565–1568 (2016)Google Scholar
  54. 54.
    Triguero, I., Peralta, D., Bacardit, J., García, S., Herrera, F.: MRPR: a MapReduce solution for prototype reduction in big data classification. Neurocomputing 150(Part A), 331–345 (2015)CrossRefGoogle Scholar
  55. 55.
    Wang, C., Karimi, S.: Parallel duplicate detection in adverse drug reaction databases with spark. In: Proceedings of the 19th International Conference on Extending Database Technology, pp. 551–562. ACM, New York (2016)Google Scholar
  56. 56.
    Wang, F., Aji, A., Vo, H.: High performance spatial queries for spatial big data: from medical imaging to GIS. SIGSPATIAL Special 6, 11–18 (2014)CrossRefGoogle Scholar
  57. 57.
    Wang, J., Wu, S., Gao, H., Li, J., Ooi, B.C.: Indexing multi-dimensional data in a cloud system. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 591–602. ACM, New York (2010)Google Scholar
  58. 58.
    Wang, K., Han, J., Tu, B., Dai, J., Zhou, W., Song, X.: accelerating spatial data processing with MapReduce. In: Proceedings of the IEEE 16th International Conference on Parallel and Distributed Systems, pp. 229–236, IEEE Computer Society, Washington, DC (2010)Google Scholar
  59. 59.
    Xavier, P., Francis, F.S.: Improvisation to the R\(^*\)-Tree kNN join principles in distributed environment. Int. J. Comput. Appl. 101, 20–24 (2014)Google Scholar
  60. 60.
    Yang, M., Zheng, L., Lu, Y., Guo, M., Li, J.: Cloud-assisted spatio-textual k nearest neighbor joins in sensor networks. In: Proceedings of the 1st International Conference on Industrial Networks and Intelligent Systems, pp. 12–17. ICST, Gent, Belgium (2015)Google Scholar
  61. 61.
    Yokoyama, T., Ishikawa, Y., Suzuki, Y.: Processing all k-nearest neighbor queries in hadoop. In: Gao, H., Lim, L., Wang, W., Li, C., Chen, L. (eds.) WAIM 2012. LNCS, vol. 7418, pp. 346–351. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-32281-5_34 CrossRefGoogle Scholar
  62. 62.
    Yu, J., Wu, J., Sarwat, M.: GeoSpark: A cluster computing framework for processing large-scale spatial data. In: Proceedings of the 23rd International Conference on Advances in Geographic Information Systems, 03–06 November 2015. Association for Computing Machinery (2015)Google Scholar
  63. 63.
    Zhang, C., Li, F., Jestes, J.: Efficient parallel kNN joins for large data in MapReduce. In: Proceedings of the 15th International Conference on Extending Database Technology, pp. 38–49. ACM, New York (2012)Google Scholar
  64. 64.
    Zhang, F., Zheng, Y., Xu, D., Du, Z., Wang, Y., Liu, R., Ye, X.: Real-time spatial queries for moving objects using storm topology. ISPRS Int. J. Geo-Inf. 5, 178 (2016)CrossRefGoogle Scholar
  65. 65.
    Zhang, H., Sun, Z., Liu, Z., Xu, C., Wang, L.: Dart: a geographic information system on hadoop. In: Proceedings of the IEEE 8th International Conference on Cloud Computing, pp. 90–97. IEEE (2015)Google Scholar
  66. 66.
    Zhang, J., Mamoulis, N., Papadias, D., Tao, Y.: All-nearest-neighbors queries in spatial databases. In: Proceedings of the 16th International Conference on Scientific and Statistical Database Management, pp. 297–306. IEEE Computer Society, Washington, DC (2004)Google Scholar
  67. 67.
    Zhang, S., Han, J., Liu, Z., Wang, K., Feng, S.: Spatial queries evaluation with MapReduce. In: Proceedings of the 8th International Conference on Grid and Cooperative Computing, pp. 287–292. IEEE Computer Society, Washington, DC (2009)Google Scholar
  68. 68.
    Zhong, Y., Han, J., Zhang, T., Li, Z., Fang, J., Chen, G.: Towards parallel spatial query processing for big spatial data. In: Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, pp. 2085–2094. IEEE Computer Society, Washington, DC (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Nikolaos Nodarakis
    • 1
    Email author
  • Angeliki Rapti
    • 1
  • Spyros Sioutas
    • 2
  • Athanasios K. Tsakalidis
    • 1
  • Dimitrios Tsolis
    • 3
  • Giannis Tzimas
    • 4
  • Yannis Panagis
    • 5
  1. 1.Computer Engineering and Informatics DepartmentUniversity of PatrasPatrasGreece
  2. 2.Department of InformaticsIonian UniversityCorfuGreece
  3. 3.Department of Cultural Heritage, Management and New TechnologiesUniversity of PatrasPatrasGreece
  4. 4.Computer and Informatics Engineering DepartmentTechnological Educational, Institute of Western GreecePatrasGreece
  5. 5.Centre of Excellence for International CourtsUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations