Advertisement

Machine Learning

, Volume 107, Issue 6, pp 1069–1094 | Cite as

Improved maximum inner product search with better theoretical guarantee using randomized partition trees

  • Omid Keivani
  • Kaushik Sinha
  • Parikshit Ram
Article
  • 200 Downloads

Abstract

Recent interest in the problem of maximum inner product search (MIPS) has sparked the development of new solutions. The solutions (usually) reduce MIPS to the well-studied problem of nearest-neighbour search (NNS). To escape the curse of dimensionality, the problem is relaxed to accept approximate solutions (that is, accept anything that is approximately maximum), and locality sensitive hashing is the approximate NNS algorithm of choice. While being extremely resourceful, these existing solutions have a couple of aspects that can be improved upon—(i) MIPS can be reduced to NNS in multiple ways and there is lack of understanding (mostly theoretical but also empirical) when to choose which reduction for best accuracy or efficiency, and (ii) when MIPS is solved via approximate NNS, translating this approximation to the MIPS solution is not straightforward. To overcome these usability issues, we propose the use of randomized partition trees (RPTs) for solving MIPS. We still reduce MIPS to NNS but utilize RPTs to solve the NNS problem. RPTs find the exact NNS solution, hence the exact MIPS solution (with high probability), avoiding the need for any translation of approximation. The theoretical properties of RPTs allow us to definitively choose the best MIPS-to-NNS reduction. The empirical properties of RPTs allow us to significantly outperform the state-of-the-art while providing unique fine-grained control over the accuracy–efficiency trade-off. For example, at 80% accuracy, RPTs are 2–5\(\times \) more efficient than the state-of-the-art. Superiority of RPT comes at the cost of high space complexity overhead which can be a severe limitation of our proposed method. To address this limitation we introduce two space efficient versions of RPTs which enjoy the same superior performance of RPT while requiring a significantly reduced space complexity overhead.

Keywords

Maximum inner product search Nearest neighbor search Random projection trees Locality sensitive hashing 

References

  1. Andoni, A., & Indyk, P. (2008). Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM, 51(1), 117–122.CrossRefGoogle Scholar
  2. Bache, K., & Lichman, M. (2013). UCI machine learning repository. http://archive.ics.uci.edu/ml.
  3. Bachrach, Y., Finkelstein, Y., Gilad-Bachrach, R., Katzir, L., Koeningstein, N., Nice, N., & Paquet, U. (2014). Speeding up the xbox recommender system using a euclidean transformation for inner-product spaces. In 8th ACM Conference on Recommender Systems.Google Scholar
  4. Cai, D. (2009). Text datasets in Matlab format. http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html.
  5. Cremonesi, P., Koren, Y., & Turrin, R. (2010). Performance of recommender algorithms on top-n recommendation tasks. In 4th ACM Conference on Recommender Systems.Google Scholar
  6. Curtin, R. R., & Ram, P. (2014). Dual-tree fast exact max-kernel search. Statistical Analysis and Data Mining, 7(4), 229–253.MathSciNetCrossRefGoogle Scholar
  7. Curtin, R. R., Ram, P., & Gray, A. G. (2013). Fast exact max-kernel search. In Proceedings of SIAM data mining.Google Scholar
  8. Dasgupta, S., & Sinha, K. (2013). Randomized partition trees for exact nearest neighbor search. In The 26th Annual Conference on Learning Theory.Google Scholar
  9. Dasgupta, S., & Sinha, K. (2015). Randomized partition trees for nearest neighbor search. Algorithmica, 72(1), 237–263.MathSciNetCrossRefzbMATHGoogle Scholar
  10. Datar, M., Immorlica, N., Indyk, P., & Mirrokni, C. S. (2004). Locality-sensitive hashing based on p-stable distributions. In The 20th ACM symposium on computational geometry.Google Scholar
  11. Dean, T., Ruzon, M., Segal, M., Shlens, J., Vijayanarasimhan, S., & Yagnik, J. (2013). Fast accurate detection of 100,000 object classes on a single machine. In IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  12. Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on pattern Analysis and Machine Intelligence, 32(9), 1627–1645.CrossRefGoogle Scholar
  13. Gionis, A., Indyk, P., & Motwan, R. (1999). Similarity search in high dimensions via hashing. In 25th International Conference on Very Large Databases.Google Scholar
  14. Hyvönen, V., Pitkänen, T., Tasoulis, S. K., Jaasaari, E., Tuomainen, R., Wang, L., Corander, J., & Roos, T. (2016). Fast nearest neighbor search through sparse random projections and voting. In IEEE International Conference on BigData.Google Scholar
  15. Indyk, P., & Motwani, R. (1998). Approximate nearest neighbors: Towards removing the curse of dimensionality. In ACM symposium on theory of computing.Google Scholar
  16. Jain, P., & Kapoor, A. (2009). Active learning for large multi-class problem. In IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  17. Jégou, H., Douze, M., & Schmid, C. (2011). Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1), 117–128.CrossRefGoogle Scholar
  18. Joachims, T. (2006). Training linear SVM in linear time. In 12th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.Google Scholar
  19. Joachims, T., Finley, T., & Yu, C.-N. J. (2009). Cutting plane training of structural SVMs. Machine Learning, 77(1), 27–59.CrossRefzbMATHGoogle Scholar
  20. Keivani, O., Sinha, K., & Ram, P. (2017). Improved inner product search with better theoretical guarantees. In International Joint Conference on Neural Network.Google Scholar
  21. Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8), 30–37.CrossRefGoogle Scholar
  22. Manjunath, B. S., & Ma, W. Y. (1996). Texture features for browsing and retrieving of large image data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1), 117–128.Google Scholar
  23. Muja, M., & Lowe, D. G. (2014). Scalable nearest neighbor algorithms for high dimensional data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11), 2227–2240.CrossRefGoogle Scholar
  24. Neyshabur, B., & Srebro, N. (2015). On symmetric and asymmetric LSHs for inner product search. In 32nd International Conference on Machine Learning.Google Scholar
  25. Ram, P., & Gray, A. G. (2012). Maximum inner-product search using cone trees. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.Google Scholar
  26. Shrivasatava, A., & Li, P. (2014). Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). In 28th Annual Conference on Neural Information Processing Systems.Google Scholar
  27. Shrivasatava, A., & Li, P. (2015). Improved asymmetric locality sensitive hashing (ALSH) for maximum inner product search (MIPS). In 31st Conference on Uncertainty in Artificial Intelligence.Google Scholar
  28. Sinha, K. (2014). LSH versus randomized partition trees: Which one to use for nearest neighbor search? In 13th International Conference on Machine Learning and Applications.Google Scholar
  29. Srebro, N., Rennie, J., & Jaaakkola, T. (2005). Maximum margin matrix factorization. In 19th Annual Conference on Neural Information Processing Systems.Google Scholar
  30. Trec interpolation. http://trec.nist.gov/pubs/trec16/appendices/measures.pdf. Accessed 14 Sept 2016.

Copyright information

© The Author(s) 2018

Authors and Affiliations

  1. 1.Department of Electrical Engineering and Computer ScienceWichita State UniversityWichitaUSA
  2. 2.Skytree Inc.AtlantaUSA

Personalised recommendations