NV-QALSH: An NVM-Optimized Implementation of Query-Aware Locality-Sensitive Hashing

Abstract

Locality-Sensitive Hashing (LSH) is a popular method for answering c-approximate nearest neighbor queries in high-dimensional spaces. Existing LSH methods are either DRAM-based ones which consume a vast amount of expensive DRAM and are time-consuming to rebuild after programs reboot, or disk-based ones such as the state-of-the-art QALSH (Query-Aware LSH), which suffers from high latency of disk I/O. In this paper, we find that the emerging non-volatile memory (NVM) can be leveraged to solve the above problems. Its economic characteristics and data durability urge us to persist most of the LSH index in NVM to reduce DRAM occupancy; and its byte-addressability and low latency contribute to fast query processing. Since QALSH uses B+-Trees as index data structures and LB-Tree is the state-of-the-art NVM-optimized B+-Tree, we first directly combine QALSH with LB-Tree to get LB-QALSH. However, LB-QALSH shows poor query performance under NVM. To fully utilize the advantages of NVM, we propose an NVM-optimized implementation of QALSH, named NV-QALSH, which is the first NVM-optimized LSH. NV-QALSH adopts three optimization designs to achieve a high query performance. Experiments show that NV-QALSH outperforms LB-QALSH with a 1.5-4.7x speedup. Furthermore, compared with the state-of-the-art DRAM-based LSH, NV-QALSH greatly reduces the DRAM occupancy and the index rebuilt time.