Abstract
This paper presents a Compressed PQ Indexing (CPQI) data structure, which realizes the further compression of sparse entries, requires only sub-linear search time, and the sparse entries are no longer stored. The proposed CPQI saves storage space and is suitable for in-memory computing for large-scale data. The CPQI employs the Minimal Perfect Hash to hash the PQ code, preserve non-null entries, and store the structure very compactly; the compressed PQ hash code index no longer stores PQ code. A sub-linear time search is implemented by combining Bloom filtering with a minimum perfect hash function.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, J., Zhang, T., Song, J., et al.: A Survey on Learning to Hash. CoRR, abs/1606.00185 (2016)
Torralba, A., Murphy, K.P., Freeman, W.T., et al.: Context-based vision system for place and objectrecognition. In: Proceedings of 9th IEEE International Conference on Computer Vision (ICCV2003), Nice, France, 14–17 October 2003, pp. 273–280 (2003)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)
Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3), 145–175 (2001)
Seidl, T., Kriegel, H.: Optimal multi-step k-Nearest neighbor search. In: Proceedings of SIGMOD 1998, Proceedings ACM SIGMOD International Conference on Management of Data, Seattle, Washington, USA, 2–4 June 1998, pp. 154–165 (1998)
Xu, H., Wang, J., Li, Z., et al.: Complementary hashing for approximate nearest neighbor search. In: Proceedings of IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, 6–13 November 2011, pp. 1631–1638 (2011)
Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975)
Fu, A.W., Chan, P.M., Cheung, Y., et al.: Dynamic vp-Tree indexing for n-Nearest neighbor search given pair-wise distances. VLDB J. 9(2), 154–173 (2000)
Indyk, P.: Nearest neighbors in high-dimensional spaces. In: Proceedings of Handbook of Discrete and Computational Geometry, pp. 877–892, 2nd edn. (2004)
Datar, M., Immorlica, N., Indyk, P., et al.: Locality-sensitive hashing scheme based on p-stable distributions. In: Proceedings of the 20th ACM Symposium on Computational Geometry, Brooklyn, New York, USA, 8–11 June 2004, pp. 253–262 (2004)
Jégou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 117–128 (2011)
Ge, T., He, K., Ke, Q., et al.: Optimized product quantization. IEEE Trans. Pattern Anal. Mach. Intell. 36(4), 744–755 (2014)
Kalantidis, Y., Avrithis, Y.S.: Locally optimized product quantization for approximate nearest neighbor search. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, 23–28 June 2014, pp. 2329–2336 (2014)
Fox, E.A., Heath, L.S., Chen, Q.F., et al.: Practical minimal perfect hash functions for large databases. Commun. ACM 35(1), 105–121 (1992)
Limasset, A., Rizk, G., Chikhi, R., et al.: Fast and scalable minimal perfect hashing for massive keysets. CoRR, abs/1702.03154 (2017)
Mitzenmacher, M., Vadhan, S.P.: Why simple hash functions work: exploiting the entropy in a data stream. In: Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2008, San Francisco, California, USA, 20–22 January 2008, pp. 746–755 (2008)
Bloom, B.H.: Space/Time trade-offs in hash coding with allowable errors. Commun. ACM 13(7), 422–426 (1970)
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant 61403060, Huaian Natural Science Foundation HAB201704, Six Talent Peaks project in Jiangsu Province under Grant 2016XYDXXJS-012, the Natural Science Foundation of Jiangsu Province under Grant BK20171267, 533 talents engineering project in Huaian under Grant HAA201738.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Shan, J., Zhang, Y., Jiang, M., Jin, C., Zhang, Z. (2019). A Fast PQ Hash Code Indexing. In: Barolli, L., Xhafa, F., Javaid, N., Enokido, T. (eds) Innovative Mobile and Internet Services in Ubiquitous Computing. IMIS 2018. Advances in Intelligent Systems and Computing, vol 773. Springer, Cham. https://doi.org/10.1007/978-3-319-93554-6_37
Download citation
DOI: https://doi.org/10.1007/978-3-319-93554-6_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-93553-9
Online ISBN: 978-3-319-93554-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)