Abstract
This paper presents an efficient preference-based ranking algorithm running in two stages. In the first stage, the algorithm learns a preference function defined over pairs, as in a standard binary classification problem. In the second stage, it makes use of that preference function to produce an accurate ranking, thereby reducing the learning problem of ranking to binary classification. This reduction is based on the familiar QuickSort and guarantees an expected pairwise misranking loss of at most twice that of the binary classifier derived in the first stage. Furthermore, in the important special case of bipartite ranking, the factor of two in loss is reduced to one. This improved bound also applies to the regret achieved by our ranking and that of the binary classifier obtained.
Our algorithm is randomized, but we prove a lower bound for any deterministic reduction of ranking to binary classification showing that randomization is necessary to achieve our guarantees. This, and a recent result by Balcan et al., who show a regret bound of two for a deterministic algorithm in the bipartite case, suggest a trade-off between achieving low regret and determinism in this context.
Our reduction also admits an improved running time guarantee with respect to that deterministic algorithm. In particular, the number of calls to the preference function in the reduction is improved from Ω(n 2) to O(nlog n). In addition, when the top k ranked elements only are required (k≪n), as in many applications in information extraction or search engine design, the time complexity of our algorithm can be further reduced to O(klog k+n). Our algorithm is thus practical for realistic applications where the number of points to rank exceeds several thousand.
Article PDF
Similar content being viewed by others
References
Agarwal, S., Graepel, T., Herbrich, R., Har-Peled, S., & Roth, D. (2005). Generalization bounds for the area under the roc curve. Journal of Machine Learning Research, 6, 393–425.
Agarwal, S., & Niyogi, P. (2005). Stability and generalization of bipartite ranking algorithms. In COLT (pp. 32–47).
Ailon, N. (2007). Aggregation of partial rankings, p-ratings and top-m lists. In SODA.
Ailon, N., Charikar, M., & Newman, A. (2005). Aggregating inconsistent information: ranking and clustering. In Proceedings of the 37th annual ACM symposium on theory of computing (pp. 684–693). Baltimore, MD, USA, May 22–24, 2005. New York: ACM.
Ailon, N., & Mohri, M. (2008). An efficient reduction of ranking to classification. In Proceedings of the 21st annual conference on learning theory (COLT 2008). Helsinki: Omnipress.
Alon, N. (2006). Ranking tournaments. SIAM Journal Discrete Mathematics, 20, 137–142.
Ariely, D., Loewenstein, G., & Prelec, D. (2008). Coherent arbitrariness: stable demand curves without stable preferences. The Quarterly Journal of Economics, 118, 73–105.
Arrow, K. J. (1950). A difficulty in the concept of social welfare. Journal of Political Economy, 58, 328–346.
Balcan, M.-F., Bansal, N., Beygelzimer, A., Coppersmith, D., Langford, J., & Sorkin, G. B. (2007). Robust reductions from ranking to classification. In COLT (pp. 604–619). Berlin: Springer.
Balcan, M.-F., Bansal, N., Beygelzimer, A., Coppersmith, D., Langford, J., & Sorkin, G. B. (2008). Robust reductions from ranking to classification. Machine Learning Journal, 72, 139–153.
Clémençon, S., & Vayatis, N. (2007). Ranking the best instances. Journal of Machine Learning Research, 8, 2671–2699.
Cohen, W. W., Schapire, R. E., & Singer, Y. (1999). Learning to order things. The Journal of Artificial Intelligence Research, 10, 243–270.
Coppersmith, D., Fleischer, L., & Rudra, A. (2006). Ordering by weighted number of wins gives a good ranking for weighted tournaments. In Proceedings of the 17th annual ACM-SIAM symposium on discrete algorithms (SODA).
Cortes, C., Mohri, M., & Rastogi, A. (2007a). An alternative ranking problem for search engines. In Proceedings of the 6th workshop on experimental algorithms (WEA 2007) (pp. 1–21). Heidelberg: Springer-Verlag.
Cortes, C., Mohri, M., & Rastogi, A. (2007b). Magnitude-preserving ranking algorithms. In Proceedings of the twenty-fourth international conference on machine learning (ICML 2007). Oregon State University, Corvallis, OR.
Cossock, D., & Zhang, T. (2006). Subset ranking using regression. In COLT (pp. 605–619).
Crammer, K., & Singer, Y. (2001). Pranking with ranking. In Advances in neural information processing systems : Vol. 14. Neural information processing systems: natural and synthetic, NIPS 2001 (pp. 641–647). Vancouver, British Columbia, Canada, December 3–8, 2001. Cambridge: MIT Press.
Freund, Y., Iyer, R. D., Schapire, R. E., & Singer, Y. (2003). An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4, 933–969.
Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36.
Hedge, R., Jain, K., Williamson, D. P., & van Zuylen, A. (2007). Deterministic pivoting algorithms for constrained ranking and clustering problems. In Proceedings of the ACM-SIAM symposium on discrete algorithms (SODA).
Hoare, C. (1961). Quicksort: Algorithm 64. Communications of the ACM, 4, 321–322.
Joachims, T. (2002). Optimizing search engines using clickthrough data. In KDD ’02: Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining (pp. 133–142). New York: ACM Press.
Kenyon-Mathieu, C., & Schudy, W. (2007). How to rank with few errors. In STOC ’07: Proceedings of the thirty-ninth annual ACM symposium on theory of computing (pp. 95–103). New York: ACM Press.
Lehmann, E. L. (1975). Nonparametrics: statistical methods based on ranks. San Francisco: Holden-Day.
Montague, M. H., & Aslam, J. A. (2002). Condorcet fusion for improved retrieval. In Proceedings of the 2002 ACM CIKM international conference on information and knowledge management (pp. 538–548). McLean, VA, USA, November 4–9, 2002. New York: ACM.
Rudin, C., Cortes, C., Mohri, M., & Schapire, R. E. (2005). Margin-based ranking meets boosting in the middle. In Learning theory, 18th annual conference on learning theory, COLT 2005 Proceedings (pp. 63–78). Bertinoro, Italy, June 27–30, 2005. Berlin: Springer.
Williamson, D. P., & van Zuylen, A. (2007). Deterministic algorithms for rank aggregation and other ranking and clustering problems. In Proceedings of the 5th workshop on approximation and online algorithms (WAOA).
Author information
Authors and Affiliations
Corresponding author
Additional information
Editors: Sham Kakade and Ping Li.
Rights and permissions
About this article
Cite this article
Ailon, N., Mohri, M. Preference-based learning to rank. Mach Learn 80, 189–211 (2010). https://doi.org/10.1007/s10994-010-5176-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-010-5176-9