Antithetic and Monte Carlo kernel estimators for partial rankings
 496 Downloads
Abstract
In the modern age, rankings data are ubiquitous and they are useful for a variety of applications such as recommender systems, multiobject tracking and preference learning. However, most rankings data encountered in the real world are incomplete, which prevent the direct application of existing modelling tools for complete rankings. Our contribution is a novel way to extend kernel methods for complete rankings to partial rankings, via consistent Monte Carlo estimators for Gram matrices: matrices of kernel values between pairs of observations. We also present a novel variancereduction scheme based on an antithetic variate construction between permutations to obtain an improved estimator for the Mallows kernel. The corresponding antithetic kernel estimator has lower variance, and we demonstrate empirically that it has a better performance in a variety of machine learning tasks. Both kernel estimators are based on extending kernel mean embeddings to the embedding of a set of full rankings consistent with an observed partial ranking. They form a computationally tractable alternative to previous approaches for partial rankings data. An overview of the existing kernels and metrics for permutations is also provided.
Keywords
Reproducing kernel Hilbert space Partial rankings Monte Carlo Antithetic variates Gram matrix1 Motivation
Permutations play a fundamental role in statistical modelling and machine learning applications involving rankings and preference data. A ranking over a set of objects can be encoded as a permutation; hence, kernels for permutations are useful in a variety of machine learning applications involving rankings such as recommender systems, multiobject tracking and preference learning. It is of interest to construct a kernel in the space of the data in order to capture similarities between datapoints and thereby influence the pattern of generalisation. A kernel input is required for the maximum mean discrepancy (MMD) twosample test (Gretton et al. 2012), kernel principal component analysis (kPCA) (Schölkopf et al. 1999), support vector machines (Boser et al. 1992; Cortes and Vapnik 1995), Gaussian processes (GPs) (Rasmussen and Williams 2006) and agglomerative clustering (Duda and Hart 1973), among others.
Our main contributions are: (1) a novel and computationally tractable way to deal with incomplete or partial rankings by first representing the marginalised kernel (Haussler 1999) as a kernel mean embedding of a set of full rankings consistent with an observed partial ranking. We then propose two estimators that can be represented as the corresponding empirical mean embeddings; (2) a Monte Carlo kernel estimator that is based on sampling independent and identically distributed rankings from the set of consistent full rankings given an observed partial ranking; (3) an antithetic variate construction for the marginalised Mallows kernel that gives a lower variance estimator for the kernel Gram matrix. The Mallows kernel has been shown to be an expressive kernel; in particular, Mania et al. (2016) show that the Mallows kernel is an example of a universal and characteristic kernel, and hence, it is a useful tool to distinguish samples from two different distributions, it achieves the Bayes risk when used in kernelbased classification/regression (Sriperumbudur et al. 2011). Jiao and Vert (2015) have proposed a fast approach for computing the Kendall marginalised kernel; however, this kernel is not characteristic (Mania et al. 2016) and hence has limited expressive power.
The resulting estimators are used for a variety of kernel machine learning algorithms in the Experiments section. In particular, we present comparative simulation results demonstrating the efficacy of the proposed estimators for an agglomerative clustering task, a hypothesis test task using the maximum mean discrepancy (MMD) (Gretton et al. 2012) and a Gaussian process classification task. For the latter, we extend some of the existing methods in the software library GPy (GPy 2012).
Since the space of permutations is an example of a discrete space, with a noncommutative group structure, the corresponding reproducing kernel Hilbert spaces (RKHS) have only recently being investigated; see Kondor et al. (2007), Fukumizu et al. (2009), Kondor and Barbosa (2010), Jiao and Vert (2015) and Mania et al. (2016). First, we provide an overview of the connection between kernels and certain semimetrics when working on the space of permutations. This connection allows us to obtain kernels from given semimetrics or semimetrics from existing kernels. We can combine these semimetricbased kernels to obtain novel, more expressive kernels which can be used for the proposed Monte Carlo kernel estimator.
2 Definitions
We first briefly introduce the theory of permutation groups. A particular application of permutations is to use them to represent rankings; in fact, there is a natural onetoone relationship between rankings of n items and permutations (Stanley 2000). For this reason, we sometimes use ranking and permutation interchangeably. In this section, we state some mathematical definitions to formalise the problem in terms of the space of permutations.
Let \(\left[ n\right] =\left\{ 1,2,\ldots ,n\right\} \) be a set of indices for n items, for some \(n \in \mathbb {N}\). Given a ranking of these n items, we use the notation \(\succ \) to denote the ordering of the items induced by the ranking, so that for distinct \(i, j \in \left[ n \right] \), if i is preferred to j, we will write \(i \succ j\). Note that for a full ranking, the corresponding relation \(\succ \) is a total order on \(\{1,\ldots , n\}\).
We now outline the correspondence between rankings on \(\left[ n \right] \) and the permutation group \(S_n\) that we use throughout the paper. In words, given a full ranking of [n], we will associate it with the permutation \(\sigma \in S_n\) that maps each ranking position \(1,\ldots ,n\) to the correct object under the ranking. More mathematically, given a ranking \(a_1 \succ \cdots \succ a_n\) of \(\left[ n \right] \), we may associate it with the permutation \(\sigma \in S_n\) given by \(\sigma (j) = a_j\) for all \(j =1,\ldots ,n\). For example, the permutation corresponding to the ranking on [3] given by \(2\succ 3\succ 1\) corresponds to the permutation \(\sigma \in S_3\) given by \(\sigma (1)=2,\sigma (2)=3,\sigma (3)=1\). This correspondence allows the literature relating to kernels on permutations to be leveraged for problems involving the modelling of ranking data.
In the next section, we first review some semimetrics on \(S_n\) because of the existing relationship between semimetrics with an additional property and kernels. We state such relationship in Theorem 1.
2.1 Metrics for permutations and properties
Definition 1
 (i)
\(d(x,y)=d(y,x)\), that is, d is a symmetric function.
 (ii)
\(d(x,y)=0\) if and only if \(x=y\).
A semimetric is a metric if it satifies:
 (iii)
\(d(x,z)\le d(x,y)+d(y,z)\) for every \(x,y, z \in \mathcal {X}\), that is, d satisfies the triangle inequality.
 (1)Spearman’s footrule$$\begin{aligned} d_f(\sigma ,\sigma ') = \sum _{i=1}^n \sigma (i)  \sigma '(i)= \Vert \sigma \sigma '\Vert _1. \end{aligned}$$
 (2)Spearman’s rank correlation$$\begin{aligned} d_{\rho }(\sigma ,\sigma ')= \sum _{i=1}^n (\sigma (i) \sigma '(i))^2= \Vert \sigma \sigma '\Vert ^2_2. \end{aligned}$$
 (3)
Hamming distance
It can also be defined as the minimum number of substitutions required to change one permutation into the other.$$\begin{aligned}d_H(\sigma ,\sigma ') = \# \{i  \sigma (i) \not = \sigma '(i) \}. \end{aligned}$$  (4)Cayley distancewhere the composition operation of the permutation group \(S_n\) is denoted by \(\circ \) and \(X_j(\sigma \circ (\sigma ')^{1})= 0\) if j is the largest item in its cycle and is equal to 1 otherwise (Irurozki et al. 2016b). It is also equal to the minimum number of pairwise transpositions taking \(\sigma \) to \(\sigma '\). Finally, it can also be shown to be equal to \(nC(\sigma \circ (\sigma ')^{1})\) where \(C(\eta )\) is the number of cycles in \(\eta \).$$\begin{aligned} d_C(\sigma , \sigma ') = \sum _{j=1}^{n1}X_j(\sigma \circ (\sigma ')^{1}), \end{aligned}$$
 (5)Kendall distancewhere \(n_d(\sigma , \sigma ')\) is the number of discordant pairs for the permutation pair \((\sigma , \sigma ')\). It can also be defined as the minimum number of pairwise adjacent transpositions taking \(\sigma ^{1}\) to \((\sigma ')^{1}\).$$\begin{aligned} d_{\tau }(\sigma , \sigma ') = n_d(\sigma , \sigma '), \end{aligned}$$
 (6)\(l_p\) distances$$\begin{aligned} d_p(\sigma , \sigma ')= & {} \left( \sum _{i=1}^n \sigma (i)  \sigma '(i)^p\right) ^{\frac{1}{p}}= \Vert \sigma \sigma '\Vert _p,\, \\&\quad \mathrm{with} \,p\ge 1. \end{aligned}$$
 (7)\( l_{\infty }\) distances$$\begin{aligned} d_{\infty }(\sigma , \sigma ') ={\mathop {_{1\le i \le n}}\limits ^{\text{ max }}}\sigma (i)  \sigma '(i)= \Vert \sigma \sigma '\Vert _{\infty }. \end{aligned}$$
Definition 2
Berlinet and ThomasAgnan (2004) and ShawerTaylor and Cristianini (2004) provide indepth treatments about Mercer kernels and reproducing kernel Hilbert spaces (RKHS); see “Appendix A” for a short overview. A useful characterisation of semimetrics of negative type is given by the following theorem, which states a connection between negativetype metrics and a Hilbert space feature representation or feature map \(\varPhi \).
Theorem 1
(Berg et al. 1984) A semimetric d is of negative type if and only if there exists a Hilbert space \(\mathcal {H}\) and an injective map \(\varPhi :\mathcal {X}\rightarrow \mathcal {H}\) such that \(\forall x,x' \in \mathcal {X}\), \(d(x,x')=\Vert \varPhi (x)\varPhi (x')\Vert _{\mathcal {H}}^2\).
In the following proposition, an explicit feature representation for the Hamming distance is introduced and we show that it is a distance of negative type.
Proposition 1
Proof
Another example is Spearman’s rank correlation, which is a semimetric of negative type since it is the square of the usual Euclidean distance (Berg et al. 1984).
Another property shared by most of the semimetrics in the examples is the following
Definition 3
This property is inherited by the distanceinduced kernel from Sect. 2.2, Example 7. This symmetry is analogous to translation invariance for kernels defined in Euclidean spaces.
2.2 Kernels for \(S_n\)
 1.The Kendall kernel (Jiao and Vert 2015) is given bywhere \(n_c(\sigma , \sigma ^\prime )\) and \( n_d(\sigma , \sigma ^\prime )\) denote the number of concordant and discordant pairs between \(\sigma \) and \(\sigma ^\prime \), respectively.$$\begin{aligned} \displaystyle {k_\tau (\sigma , \sigma ^\prime ) = \frac{n_c(\sigma , \sigma ^\prime )  n_d(\sigma , \sigma ^\prime )}{\left( {\begin{array}{c}d\\ 2\end{array}}\right) }}, \end{aligned}$$
 2.The Mallows kernel (Jiao and Vert 2015) is given by$$\begin{aligned} \displaystyle {k_{\lambda }(\sigma , \sigma ^\prime ) = \exp (\lambda n_d(\sigma , \sigma ^\prime ))}. \end{aligned}$$
 3.The Polynomial kernel of degree m (Mania et al. 2016) is given by$$\begin{aligned} \displaystyle { k_{P}^{(m)}(\sigma , \sigma ^\prime ) = (1 + k_{\tau }(\sigma , \sigma ^\prime ))^m}. \end{aligned}$$
 4.The Hamming kernel is given by$$\begin{aligned} \displaystyle {k_H(\sigma , \sigma ^\prime ) = \text {Trace}\left[ \left( \varPhi (\sigma )\varPhi (\sigma '\right) ^T\right] }. \end{aligned}$$
 5.An exponential semimetric kernel is given bywhere d is a semimetric of negative type.$$\begin{aligned} \displaystyle {k_{\text{ exp }}(\sigma , \sigma ^\prime ) = \exp \left\{ \lambda d(\sigma , \sigma ^\prime )\right\} }, \end{aligned}$$
 6.The diffusion kernel (Kondor and Barbosa 2010) is given bywhere \(\beta \in \mathbb {R}\) and q is a function that must satisfy \(q(\pi )=q(\pi ^{1})\) and \(\sum _{\pi }q(\pi )=0\). A particular case is \(q(\sigma ,\sigma ')=1\) if \(\sigma \) and \(\sigma '\) are connected by an edge in some Cayley graph representation of \(S_n\), and \(q(\sigma ,\sigma ')=\text {degree}_{\sigma }\) if \(\sigma =\sigma '\) or \(q(\sigma ,\sigma ')=0\) otherwise.$$\begin{aligned} \displaystyle {k_{\beta }(\sigma , \sigma ^\prime ) = \exp \left\{ \beta q(\sigma \circ \sigma ^\prime )\right\} }, \end{aligned}$$
 7.The semimetric or distanceinduced kernel (Sejdinovic et al. 2013): if the semimetric d is of negative type, then, a family of kernels k, parameterised by a central permutation \(\sigma _0\), is given by$$\begin{aligned} \displaystyle k_d(\sigma ,\sigma ')= \frac{1}{2}\left[ d(\sigma ,\sigma _0)+d(\sigma ',\sigma _0)d(\sigma ,\sigma ')\right] . \end{aligned}$$
In the case of the symmetric group of degree n, \(S_n\), there exist kernels that are right invariant, as defined in Equation (4). This invariance property is useful because it is possible to write down the kernel as a function of a single argument and then obtain a Fourier representation. The caveat is that this Fourier representation is given in terms of certain matrix unitary representations due to the nonAbelian structure of the group (James 1978). Even though the space is finite, and every irreducible representation is finitedimensional (Fukumizu et al. 2009), these Fourier representations do not have closedform expressions. For this reason, it is difficult to work on the spectral domain in contrast to the \(\mathbb {R}^n\) case. There is also no natural measure to sample from such as the one provided by Bochner’s theorem in Euclidean spaces (Wendland 2005). In the next section, we will present a novel Monte Carlo kernel estimator for the case of partial rankings data.
3 Partial rankings
Having provided an overview of kernels for permutations, and reviewed the link between permutations and rankings of objects, we now turn to the practical issue that in real data sets, we typically have access only to partial ranking information, such as pairwise preferences and topk rankings. Partial rankings can be obtained from pairwise comparisons data given certain assumptions. For instance, a classic generative model for pairwise comparisons that can be used to obtain topk rankings is the Bradley–Terry model (Bradley and Terry 1952) and its extension to multiple comparisons, the Plackett–Luce model (Luce 1959; Plackett 1974). See (Chen et al. 2017) for details on how to obtain a topk partial ranking given pairwise comparisons from the Bradley–Terry model and (Caron et al. 2014) for a nonparametric Bayesian extension of the Plackett–Luce model and references therein. In the following, as Jiao and Vert (2015), we assume that our data are partial rankings of the following types
Definition 4
(Exhaustive partial rankings, topkrankings) Let \(n \in \mathbb {N}\). A partial ranking on the set [n] is specified by an ordered collection \(\varOmega _1 \succ \cdots \succ \varOmega _l\) of disjoint nonempty subsets \(\varOmega _1,\ldots ,\varOmega _l \subseteq [n]\), for any \(1 \le l \le n\). The partial ranking \(\varOmega _1 \succ \cdots \succ \varOmega _l\) encodes the fact that the items in \(\varOmega _i\) are preferred to those in \(\varOmega _{i+1}\), for \(i=1,\ldots ,l1\). A partial ranking \(\varOmega _1 \succ \cdots \succ \varOmega _l\) with \(\cup _{i=1}^l \varOmega _i = [n]\) termed exhaustive, as all items in [n] are included within the preference information. A topk partial ranking is a particular type of exhaustive ranking \(\varOmega _1 \succ \cdots \succ \varOmega _{l}\), with \(\varOmega _1 = \cdots = \varOmega _{l1} = 1\), and \(\varOmega _{l} = [n] \setminus \cup _{i=1}^{l1} \varOmega _i\). We will frequently identify a partial ranking \(\varOmega _1 \succ \cdots \succ \varOmega _l\) with the set \(R(\varOmega _1,\ldots ,\varOmega _l) \subseteq S_n\) of full rankings consistent with the partial ranking. Thus, \(\sigma \in R(\varOmega _1,\ldots ,\varOmega _l)\) iff for all \(1\le i< j \le l\), and for all \(x \in \varOmega _i, y \in \varOmega _j\), we have \(\sigma ^{1}(x) < \sigma ^{1}(y)\). When there is potential for confusion, we will use the term “subset partial ranking” when referring to a partial ranking as a subset of \(S_n\), and “preference partial ranking” when referring to a partial ranking with the notation \(\varOmega _1 \succ \cdots \succ \varOmega _l\).
We propose a variety of Monte Carlo methods to estimate the marginalised kernel of Eq. (5) for the general case, where direct calculation is intractable.
Definition 5

For each \(i=1\ldots ,I\), the permutations \((\sigma ^{(i)}_m)_{m=1}^{M_i}\) are drawn exactly from the distribution \(p(\cdot R_i)\). In this case, the weights are simply \(w^{(i)}_n = 1\) for \(m=1,\ldots ,M_i\).

For each \(i=1,\ldots ,I\), the permutations \((\sigma ^{(i)}_m)_{m=1}^{M_i}\) drawn from some proposal distribution \(q(\cdot R_i)\) with the weights given by the corresponding importance weights\(w^{(i)}_n = p(\sigma ^{(i)}_nR) / q(\sigma ^{(i)}_nR)\) for \(m=1,\ldots ,M_i\).
An alternative perspective on the estimator defined in Eq. (7), more in line with the literature on random feature approximations of kernels, is to define a random feature embedding for each of the partial rankings \((R_i)_{i=1}^I\).
Theorem 2
Proof
Note that the RKHS in which these embeddings take values is finitedimensional, and the Monte Carlo estimator is the average of iid terms, each of which is equal to the true embedding in expectation. Thus, we immediately obtain unbiasedness of the Monte Carlo embedding. \(\square \)
Theorem 3
The Monte Carlo kernel estimator from Eq. (7) does define a positivedefinite kernel; further, it yields unbiased estimates of the offdiagonal elements and consistent for the diagonal elements of the kernel matrix.
Proof
We first deal with the positivedefiniteness claim. Let \(R_1, \ldots , R_I \subseteq S_n\) be a collection of partial rankings, and for each \(i=1,\ldots ,I\), let \((\sigma ^{(i)}_{m}, w^{(i)}_{m})_{m=1}^{M_i}\) be an i.i.d. weighted collection of complete rankings distributed according to \(p(\cdot  R_i)\). To show that the Monte Carlo kernel estimator \(\widehat{K}\) is positive definite, we observe that by Eq. (8), the \(I \times I\) matrix with (i, j)th element given by \(\widehat{K}(R_i, R_j)\) is the Gram matrix of the vectors \((\widehat{\varvec{\varPhi }}(R_i))_{i=1}^I\) with respect to the inner product of the Hilbert space \(\mathcal {H}_K\). We therefore immediately deduce that the matrix is positive semidefinite. Furthermore, the Monte Carlo kernel estimator is unbiased for the offdiagonal elements and consistent for the diagonal elements of the kernel matrix; see Appendix C in the supplementary material for the proof. \(\square \)
We highlight that whilst the mean embedding estimator in Eq. (9) is unbiased, the corresponding kernel estimator is consistent for the diagonal elements of the kernel matrix and unbiased for the offdiagonal elements. Having established that the Monte Carlo estimator \(\widehat{K}\) is itself a kernel, we note that when it is evaluated at two partial rankings \(R, R^\prime \subseteq S_n\), the resulting expression is not a sum of iid terms; the following result quantifies the quality of the estimator through its variance.
Theorem 4
The proof is given in the supplementary material, “Appendix D”. We have presented some theoretical properties of the embedding corresponding to the Monte Carlo kernel estimator which confirm that it is a sensible embedding. In the next section, we present a lower variance estimator based on a novel antithetic variates construction.
4 Antithetic random variates for permutations
A common, computationally cheap variancereduction technique in Monte Carlo estimation of expectations of a given function is to use antithetic variates (Hammersley and Morton 1956), the purpose of which is to introduce negative correlation between samples without affecting their marginal distribution, resulting in a lower variance estimator. Antithetic samples have been used when sampling from Euclidean vector spaces, for which antithetic samples are straightforward to define. Ross (2006) defines the antithetic of a full ranking by reversing the order of the original permutation. We give a definition of antithetic permutations for partial rankings in terms of distance maximisation and show that this coincides with the definition of Ross (2006) in the case of full rankings. We begin with a preliminary lemma, before giving the full definition of antithetic permutations given a fixed partial ranking.
Lemma 1
See “Appendix B” for the proof.
Definition 6
(Antithetic permutations) Let \(R \subseteq S_n\) be a topk partial ranking. The antithetic operator \(A_{R} : R \rightarrow R\) maps each permutation \(\sigma \in R\) to the permutation in R of maximal Kendall distance from \(\sigma \). \(A_R(\sigma )\) is said to be antithetic to \(\sigma \).
This definition of antithetic samples for permutations has parallels with the standard notion of antithetic samples in vector spaces, in which typically a sampled vector \(x \in \mathbb {R}^d\) is negated to form \(x\), its antithetic sample; \(x\) is the vector maximising the Euclidean distance from x, under the restrictions of fixed norm. We note here also that the computational cost of generating an antithetic permutation via the method described in Lemma 1 is no greater than the cost associated with generating an independent permutation.
Proposition 1
Let R be a partial ranking and \(\left\{ \sigma ,A_{R}({\sigma })\right\} \) be an antithetic pair from R, \(\sigma \) is distributed uniformly in the region R. Let \(d_{\tau }:S_n\rightarrow \mathbb {R}^{+}\) be the Kendall distance and \(\sigma _0\in R\) a fixed permutation, let \(X=d_{\tau }(\sigma ,\sigma _0)\) and \( Y= d_{\tau }(A_{R}({\sigma }),\sigma _0)\), then X and Y have negative covariance.
Proposition 1 is useful because one of the main tasks in statistical inference is to compute expectations of a function of interest, denoted by h. Once the antithetic variates are constructed, the functional form of h determines whether or not the antithetic variate construction effectively produces a lower variance estimator for its expectation. The proof of this proposition is presented after the relevant lemmas are proved. If h is a monotone function, we have the following corollary.
Corollary 2
Let h be a monotone increasing (decreasing) function. Then, the random variables \(h\left( X\right) \) and \(h\left( Y\right) \) have negative covariance.
Proof
The random variable Y from Proposition 1 is equal in distribution to \(Y{\mathop {=}\limits ^{d}}CX\), where C is a constant which specialises depending on whether \(\sigma \) is a full ranking or an exhaustive partial ranking; see the proof of Proposition 1 in the next section for the specific form of the constant for each case. By Chebyshev’s integral inequality (Fink and Jodeit 1984), the covariance between a monotone increasing (decreasing) and a monotone decreasing (increasing) functions is negative. \(\square \)
The next theorem presents the antithetic empirical feature embedding and corresponding antithetic kernel estimator. Indeed, if we take the inner product between two embeddings, this yields the kernel antithetic estimator which is a function of a pair of partial rankings subsets. In this case, the h function from above is the kernel evaluated in each pair, and this is an example of a Ustatistic (Serfling 1980, Chapter 5).
Theorem 5
\((\sigma ^{(j)}_n, A_{R_j}(\sigma ^{(j)}_n))_{n=1}^{N}\), from \(R_j\).
Proof
Since the antithetic kernel embedding is a convex combination of the Monte Carlo kernel embedding, unbiasedness follows. \(\square \)
In the next section, we present the main result about the kernel estimator from Eq. (10), namely, that it has lower asymptotic variance than the Monte Carlo kernel estimator from Eq. 7 if we use the Mallows kernel.
4.1 Variance of the antithetic kernel estimator
We now establish some basic theoretical properties of antithetic samples in the context of marginalised kernel estimation. In order to do so, we require a series of lemmas to derive the main result in Theorem 6 that guarantees that the antithetic kernel estimator has lower asymptotic variance than the Monte Carlo kernel estimator for the marginalised Mallows kernel.
The following result shows that antithetic permutations may be used to achieve coupled samples which are marginally distributed uniformly on the subset of \(S_n\) corresponding to a topk partial ranking.
Lemma 2
If \(R \subseteq S_n\) is a topk partial ranking, then if \(\sigma \sim \text {Unif}(R)\), then \(A_{R}(\sigma ) \sim \text {Unif}(R)\).
See “Appendix B” for the proof. Lemma 2 establishes a base requirement of an antithetic sample—namely, that it has the correct marginal distribution. In the context of antithetic sampling in Euclidean spaces, this property is often trivial to establish, but the discrete geometry of \(S_n\) makes this property less obvious. Indeed, we next demonstrate that the condition of exhaustiveness of the partial ranking in Lemma 2 is necessary.
Example 1
We further show that the condition of right invariance of the metric d is necessary in the next example.
Example 2
Examples 1 and 2 serve to illustrate the complexity of antithetic sampling constructions in discrete spaces. Finally, we remark that an alternative phrasing of Lemma 2 is that the pushforward of the distribution \(\text {Unif}(R)\) through the function \(A_R\) is again \(\text {Unif}(R)\). Whilst it may be possible to design distributions such that \(p(\cdot R)\) has this property for each topk ranking \(R \subseteq S_n\), many commonly used nonuniform distributions over permutations, such as Mallows models, do not satisfy this property.
We now begin direct calculation with antithetic permutations and partial rankings. We primarily focus on the case of topk rankings, as calculation turns out to be particularly tractable in this case and also due to the fact that topk rankings feature in many applications of interest. The following two lemmas state some useful relationships between the distance between two permutations \((\sigma ,\nu )\) and the corresponding pair \((A_{R}({\sigma }),\nu )\) in both the unconstrained and constrained cases which correspond to not having any partial ranking information and having partial ranking information, respectively.
Lemma 3
Let \(\sigma , \nu \in S_n\). Then, \(d_{\tau }(\sigma , \nu ) = \)\(\left( {\begin{array}{c}n\\ 2\end{array}}\right)  d_{\tau }(A_{S_n}(\sigma ), \nu )\).
Proof
This is immediate from the interpretation of the Kendall distance as the number of discordant pairs between two permutations; a distinct pair \(i,j \in [n]\) is discordant for \(\sigma , \nu \) iff they are concordant for \(A_{S_n}(\sigma ), \nu \). \(\square \)
In fact, Lemma 3 generalises in the following manner.
Lemma 4
Let R be a topk ranking \(a_1 \succ \cdots \succ a_l \succ [n] \setminus \{a_1, \ldots , a_l\}\), and let \(\sigma , \nu \in R\). Then \(d_{\tau }(\sigma , \nu ) = \left( {\begin{array}{c}nl\\ 2\end{array}}\right)  d_{\tau }(A_{R}(\sigma ), \nu )\).
See “Appendix B” for the proof. Next, we show that it is possible to obtain a unique closest element in a given partial ranking set R, denoted by \(\varPi _R(\nu )\), with respect to any given permutation \(\nu \in S_n,\nu \notin R\). This is based on the usual generalisation of a distance between a set and a point (Dudley 2002). We then use such closest element in Lemmas 6 and 7 to obtain useful decompositions of distances identities. Finally, in Lemma 8 we verify that the closest element is also distributed uniformly on a subset of the original set R.
Lemma 5
Let \(R \subseteq S_n\) be a topk partial ranking, let \(\nu \in S_n\) be arbitrary. There is a unique closest element in R to \(\nu \). In other words, \(\arg \min _{\sigma \in R}d_{\tau }(\sigma , \nu )\) is a set of size 1.
See “Appendix B” for the proof.
Definition 7
Let \(R \subseteq S_n\) be a topk partial ranking. Let \(\varPi _R : S_n \rightarrow R\) be the map that takes a permutation to the corresponding Kendallclosest permutation in R; by Lemma 5, this is well defined.
Lemma 6
See “Appendix B” for the proof.
Lemma 7
See “Appendix B” for the proof.
Lemma 8
See “Appendix B” for the proof.
Having introduced the antithetic operator for a topk partial ranking R, \(A_R : R \rightarrow R\) and the projection map \(\varPi _R : S_n \rightarrow R\), we next study how these operations interact with one another.
Lemma 9
See “Appendix B” for the proof.
Finally, the last lemma states the most general identity for a distance, which involves the antithetic operator, the closest element map given a partial rankings set R and a subset of it, denoted by \(R''\).
Lemma 10
See “Appendix B” for the proof.
Proof of Proposition 1
\(d_{\tau }(A_{R}({\sigma }),\sigma _0)={n \atopwithdelims ()2}d_{\tau }(\sigma ,\sigma _0), \forall \sigma \in S_n\), \(\forall n \in \mathbb {N}\) by Lemma 3.
Case\(\mathbf {\emptyset \subset R}\): Let \(\sigma _0\in R\), we have that
\(d_{\tau }(A_{R}({\sigma }),\sigma _0)={nk\atopwithdelims ()2}\)\(d_{\tau }(\sigma ,\sigma _0)\)\(\forall \sigma _0\in R\) by Lemma 4. \(\square \)
In general, if \(\sigma _0\notin R\), by Lemma 7, \(d_{\tau }(A_{R}({\sigma }),\sigma _0)=\)
\(d_{\tau }(\sigma , \sigma _0) + \left( {\begin{array}{c}nk\\ 2\end{array}}\right)  2d_{\tau }(\sigma , \varPi _{R_i}(\sigma _0))\).
After proving all the relevant Lemmas, we now present our main result regarding antithetic samples, namely, that this scheme provides negatively correlated pairs of samples.
Theorem 6
Consider the antithetic kernel estimator for the Mallows kernel evaluated on a pair of partial rankings \(R_i, R_j\) using M antithetic pairs of samples \((\sigma ^{(i)}_m, A_{R_i}(\sigma ^{(i)}_m))_{m=1}^{M}\) from region \(R_i\) and N antithetic pairs of samples \((\sigma ^{(j)}_n, A_{R_j}(\sigma ^{(j)}_n))_{n=1}^{N}\), from \(R_j\). The asymptotic variance of this estimator is lower than the kernel estimator using 2M (respectively, 2N) i.i.d. samples from \(R_i\) (respectively, \(R_j\)).
Proof
We remark that in case (i), the 16 terms that appear in the summand all have the same distribution in the antithetic and i.i.d. case, so terms of the form (i) contribute no difference between antithetic and i.i.d.. There are \(\mathcal {O}(N^2 M + M^2 N)\) terms of the form (ii) and \(\mathcal {O}(NM)\) terms of the form (iii). We thus refer to terms of the form (ii) as cubic terms and terms of the form (iii) as quadratic terms. We observe that due to the proportion of cubic terms to quadratic terms diverging as \(N,M \rightarrow \infty \), it is sufficient to prove that each cubic term is less in the antithetic case than the i.i.d. case to establish the claim of lower MSE.

Each of the terms \(d_{\tau }(\varPi _{R_1}(\nu _{m}), \nu _{m})\) and
\(d_{\tau }(\varPi _{R_1}(\nu _{m^\prime }), \nu _{m^\prime })\) has the same distribution under the i.i.d. case and antithetic case. Further, in both cases, \(d_{\tau }(\varPi _{R_1}(\nu _{m}), \nu _{m})\) is independent of \(\varPi _{R_1}(\nu _m)\), and \(d_{\tau }(\varPi _{R_1}(\nu _{m^\prime }), \nu _{m^\prime })\) is independent of \(\varPi _{R_1}(\nu _{m^\prime })\), so these two terms are independent of all others appearing in the sum in both cases.

Each of the terms \(d_{\tau }(\varPi _{R_3}(\sigma _n), \varPi _{R_1}(\nu _m))\) and
\(d_{\tau }(\varPi _{R_3}(\widetilde{\sigma }_n), \varPi _{R_1}(\nu _{m^\prime }))\) has the same distribution under the i.i.d. case and the antithetic case and is independent of all other terms in both cases.

We deal with the terms \(d_{\tau }(\sigma _n, \varPi _{R_3}(\sigma _n))\) and
\(d_{\tau }(\widetilde{\sigma }_n, \varPi _{R_3}(\widetilde{\sigma }_n))\) using Lemma 10. More specifically, under the i.i.d. case, these two distances are clearly i.i.d.. However, under the antithetic case, the lemma tells us that the sum of these two distances is equal to the mean under the distribution of the i.i.d. case almost surely. Thus, in the antithetic case, this random variable has the same mean as in the i.i.d. case, but is more concentrated (strictly so iff \(d(\sigma _n, \varPi _{R_3}(\sigma _n))\) is not a constant almost surely, which is the case iff \(R_1 \not = R_3\)).
4.2 The antithetic kernel estimator and kernel herding
In this section, having established the variancereduction properties of antithetic samples in the context of Monte Carlo kernel estimation, we now explore connections to kernel herding (Chen et al. 2010). Kernel herding is a deterministic approach to numerical integration, in which quadrature points are selected according to a distanceminimisation algorithm taking place in a particular Hilbert space.
Theorem 7
The antithetic variate construction of Theorem 5 is equivalent to the optimal solution for the first two steps of a kernel herding procedure in the space of permutations.
Proof
After this result, one would like to do a herding procedure for more than two steps. However, the solution is not the same as picking k herding samples simultaneously. Specifically, the following counterexample, illustrated in Fig. 4, clearly shows why. The left plot shows the result of solving the herding objective for 2 samples—the result is an antithetic pair of samples for the region R. If a third sample is selected greedily, with these first two samples fixed, it will yield a different result than if the herding objective is solved for 3 samples simultaneously, as illustrated in the right of the figure.
Remark 3
Theorem 7 says that if we first pick a point uniformly at random from R, then put it into the herding objective and then select the second deterministically to minimise the herding objective and this is equivalent to the antithetic variate construction of Definition 6. Alternatively, we could pick the second point uniformly at random from R, independently from the first point. This second scheme will produce a higher value of the herding objective on average.
After the two estimators for kernel matrices have been constructed, we use them in some experiments to assess their performance in the next section.
5 Experiments
Tree purities for the sushi data set using a subsample of 100 users with the full Gram matrix K, a censored data set of \(topk=4\) partial rankings for the vanilla Monte Carlo estimator \(\widehat{K}\) and the antithetic Monte Carlo estimator \(\widehat{K}^{a}\), with \(n_{mc}=20\) Monte Carlo samples
Kendall  Mallows  Semiexp Hamming  Semiexp Cayley  Semiexp Spearman  

K Average  0.83  0.75  0.81  0.72  0.81 
\(\widehat{K}\) Average  0.78 (0.052)  0.79 (0.058)  0.79 (0.063)  0.82 (0.040)  0.78 (0.062) 
\(\widehat{K}^{a}\) Average  NA  0.77 (0.050)  NA  NA  NA 
Definition 6 states the antithetic permutation construction with respect to a given permutation for Kendall’s distance. In order to consider partial rankings data, we should respect the observed preferences when obtaining the antithetic variate. Algorithm 1 describes how to sample an antithetic permutation and simultaneously respect the constraints imposed by the observed partial ranking. Namely, the antithetic permutation has the observed preferences fixed in the same locations as the original permutation and only reverses the unobserved locations. This corresponds to maximising the Kendall distance between the permutation pair whilst respecting the constraints and ensures that both permutations have the right marginals as stated in Lemmas 1 and 2.
5.1 Data sets
Synthetic data set The synthetic data set for the nonparametric hypothesis test experiment, where the null hypothesis is \(H_0:P=Q\) and the alternative is \(H_1:P\ne Q\), is the following: the data set from the P distribution is a mixture of Mallows distributions (Diaconis 1988) with the Kendall and Hamming distances. The central permutations are given by the identity permutation and the reverse of the identity, respectively, with lengthscale equal to one. The data set from the Q distribution is a sample from the uniform distribution over \(S_n\), where \(n = 6\).
Sushi data set This data set contains rankings about sushi preferences given by 5000 users (Kamishima et al. 2009). The users ranked 10 types of sushi, and the labels correspond to the user’s region This data set is used for the Gaussian and ten for the agglomerative clustering task.
5.2 Agglomerative clustering
In Table 1, the true and estimated purities using the full rankings and the partial rankings data sets are reported. We assumed that the true labels are given by the user’s region, and there are ten different possible regions. The true purity corresponds to an agglomerative clustering algorithm using the Gram matrix obtained from the full rankings. We can compute the Gram matrix for the full rankings because we have access to all of the users’ rankings over the ten different types of sushi. The antithetic Monte Carlo estimator outperforms the vanilla Monte Carlo estimator in terms of average purity since it is closer to the true purity. It also has a lower standard deviation when estimating the marginalised Mallows kernel.
5.3 Nonparametric hypothesis test with MMD
Standard deviations for p values computed with the Monte Carlo and antithetic estimators
# obs  10  15  20  25  30  35  40 

Monte Carlo  0.0853  0.0910  0.0830  0.1109  0.0677  0.0596  0.0236 
Antithetic  0.0706  0.0663  0.0712  0.0594  0.0502  0.0363  0.0222 
Averaged over 10 runs with 4 Monte Carlo samples per run, \(n=10,topk=6\)
Test accuracy  Train aveloglik  Test aveloglik  

Mallows  
Full model  0.9  \(\) 0.2070  \(\) 0.5457 
MC  0.74  \(\) 0.2486(0.005)  \(\) 0.563(0.020) 
Antithetic  0.75  \(\) 0.262(0.001)  \(\) 0.573(0.002) 
Gaussian  
Full model  0.75  \(\) 0.2215  \(\) 0.7014 
MC  0.72  \(\) 0.2890(0.0245)  \(\) 0.5737(0.043) 
Antithetic  NA  NA  NA 
Kendall  
Full model  0.7  \(\) 0.311(3.01\(\times 10^{\,6}\))  \(\) 0.597(3.5\(\times 10^{\,6}\)) 
MC  0.66  \(\) 0.3575(0.008)  \(\) 0.7063(0.052) 
Antithetic  NA  NA  NA 
5.4 Gaussian process classifier
In this experiment, two different kernels were used to compute the estimators for the Gram matrix between different pairs of partial rankings subsets. The matrix was then provided as the input to a Gaussian process classifier (Neal 1998). The Python library GPy (2012) was extended with custom kernel classes for partial rankings which compute both the Monte Carlo and antithetic kernel estimators for partial rankings subsets. Previously, it was only possible to do pointwise evaluations of kernels, but our implementation allows to compute the kernels over pairs of partial ranking subsets by storing the sets in a tensor first.
We used the sushi data set from Sect. 5.1 with the labels binarised in East Japan or West Japan regions. We selected a random subset of the observations of size 100 and used 80%, for the training set and 20% for the test set. In the Mallows kernel case, we used the median distance heuristic (Takeuchi et al. 2006; Schölkopf and Smola 2002) with the Kendall distance to compute the bandwidth parameter and a scale parameter of 9.5. We performed a grid search over different values of the scale parameter and picked the one that had the largest classification accuracy for the test set.
In Table 3, the results of running the Gaussian process classifier are reported using the marginalised Mallows kernel, the marginalised Gaussian kernel and the marginalised Kendall kernel as well as the corresponding estimators. Since the Mallows kernel is based on the Kendall distance, it is a kernel specifically tailored for permutations and it is the best in terms of predictive performance. In contrast, the Gaussian kernel is a kernel that is suitable for Euclidean spaces and it does not take into account the data type, and it still exhibits good predictive performance. The Kendall kernel does take into account the data type; however, it performs the worst. The full model corresponds to using the Gram matrix computed with the full rankings, and MC and antithetic refer to the Gram matrix obtained with the Monte Carlo and antithetic kernel estimators. We observe that the test and train loglikelihoods obtained with the antithetic kernel estimator have lower variance as expected.
6 Conclusion
We addressed the problem of extending kernels to partial rankings by introducing a novel Monte Carlo kernel estimator and explored variancereduction strategies via an antithetic variates construction. Our schemes lead to a computationally tractable alternative to previous approaches for partial rankings data. The Monte Carlo scheme can be used to obtain an estimator of the marginalised kernel with any of the kernels reviewed herein. The antithetic construction provides an improved version of the kernel estimator for the marginalised Mallows kernel. Our contribution is noteworthy because the computation of most of the marginalised kernels grows superexponentially with respect to the number of elements in the collection; hence, it quickly becomes intractable for relatively small values of the number of ranked items n. An exception is the fast approach for computing the convolution kernel proposed by Jiao and Vert (2015), which is only valid for Kendall kernel. Mania et al. (2016) have showed that the Kendall kernel is not characteristic using noncommutative Fourier analysis to show that it has a degenerate spectrum. For this reason, using other kernels for permutations might be desirable depending on the task at hand.
One possible direction for future work includes the use of explicit feature representations for traditional random features schemes to further reduce the computational cost of the Gram matrix. Another possible application is to use our method with pairwise preference data where users are not necessarily consistent about their preferences. In this type of data, we could still extract a partial ranking from a given user, then sample from the space of the corresponding full rankings consistent with this observed partial ranking and obtain our Monte Carlo kernel estimator. This would benefit from our framework because having a partial ranking is in general more informative that having pairwise comparisons or star ratings.
Another natural direction for future work is to develop variancereduction sampling techniques for a wider variety of kernels over permutations, and to the extent the theoretical analysis of these constructions to discrete graphs more generally.
Notes
Acknowledgements
We thank the anonymous reviewers for their valuable comments which have improved the quality of our manuscript and Ryan Adams, for insightful discussions. Maria Lomeli and Zoubin Ghahramani acknowledge support from the Alan Turing Institute (EPSRC Grant EP/N510129/1), EPSRC Grant EP/N014162/1, and donations from Google and Microsoft Research. Arthur Gretton thanks the Gatsby Charitable Foundation for financial support. Mark Rowland acknowledges support by EPSRC Grant EP/L016516/1 for the Cambridge Centre for Analysis.
Supplementary material
References
 Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 68(3), 337–404 (1950)MathSciNetCrossRefzbMATHGoogle Scholar
 Berg, C., Christensen, J.P.R., Ressel, P.: Harmonic Analysis on Semigroups. Springer, New York (1984)CrossRefzbMATHGoogle Scholar
 Berlinet, A., ThomasAgnan, C.: Reproducing Hilbert Spaces for Probability and Statistics. Kluwer Academic Publishers, Dordrecht (2004)CrossRefzbMATHGoogle Scholar
 Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: ACM workshop on Computational Learning Theory (1992)Google Scholar
 Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs. Biometrika 39, 324–345 (1952)MathSciNetzbMATHGoogle Scholar
 Busse, L.M., Orbanz, P., Buhmann, J.M.: Cluster analysis of heterogeneous rank data. In: International Conference on Machine Learning (2007)Google Scholar
 Caron, F., Teh, Y.W., Murphy, T.B.: Bayesian nonparametric Plackett–Luce models for the analysis of preferences for college degree programmes. Ann Appl Stat 8, 1145–1181 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
 Chen, Y., Fan, J., Ma, C., Wang, K.: Spectral method and regularized MLE are both optimal for topk ranking (2017). arXiv:1707.09971
 Chen, Y., Welling, M., Smola, A.: SuperSamples from Kernel Herding. In: UAI (2010)Google Scholar
 Chierichetti, F., Dasgupta, A., Haddadan, S., Kumar, R., Lattanzi, S.: Mallows models for topk lists. In: Advances in Neural Information Processing Systems (2018)Google Scholar
 Cortes, C., Vapnik, V.N.: Supportvector networks. J. Mach. Learn. 20, 273–297 (1995)zbMATHGoogle Scholar
 Deza, M.M., Deza, E.: Encyclopedia of Distances. Springer, Berlin, Heidelberg (2009)CrossRefzbMATHGoogle Scholar
 Diaconis, P.: Group Representations in Probability and Statistics. Institute of Mathematical Statistics lecture notes (1988)Google Scholar
 Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analysis. Wiley, New York (1973)zbMATHGoogle Scholar
 Dudley, R.: Real Analysis and Probability. Cambridge University Press, Cambridge (2002)CrossRefzbMATHGoogle Scholar
 Fernández, V.A., Gamero, M.D.J., García, J.M.: A test for the twosample problem based on empirical characteristic functions. Comput. Stat. Data Anal. 52, 3730–3748 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
 Fink, A.M., Jodeit, M.: On Chebyshev’s other inequality. Inequal. Stat. Probab. 5, 115–120 (1984)MathSciNetCrossRefGoogle Scholar
 Fligner, M.A., Verducci, J.S.: Distance based ranking models. J R Stat Soc Ser B 48, 359–369 (1986)MathSciNetzbMATHGoogle Scholar
 Fukumizu, K., Sriperumbudur, B., Gretton, A., Schölkopf, B.: Characteristic kernels on groups and semigroups. In: Neural Information processing systems (2009)Google Scholar
 GPy. since 2012. GPy: A Gaussian process framework in python. http://github.com/SheffieldML/GPy
 Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A Kernel two sample test. J. Mach. Learn. Res. 13, 723–773 (2012)MathSciNetzbMATHGoogle Scholar
 Hammersley, J.M., Morton, K.W.: A new Monte Carlo technique: antithetic variates. Math. Proc. Camb. Philos. Soc. 52, 449–475 (1956)MathSciNetCrossRefzbMATHGoogle Scholar
 Haussler, D.: Convolution Kernels on Discrete Structures (1999)Google Scholar
 Irurozki, E., Calvo, B., Lozano, J.A.: PerMallows: an R package for Mallows and Generalized Mallows models. J. Stat. Softw. (2016a)Google Scholar
 Irurozki, E., Calvo, B., Lozano, J.A.: Sampling and learning Mallows and Generalized Mallows models under the Cayley distance. Methodol. Comput. Appl. Probab. 20, 1–35 (2016b)MathSciNetCrossRefzbMATHGoogle Scholar
 James, G.D.: The Representation Theory of the Symmetric Groups. Springer, Berlin (1978)CrossRefzbMATHGoogle Scholar
 Jiao, Y., Vert, J.P.: The Kendall and Mallows Kernels for permutations. In: International conference for Machine learning (2015)Google Scholar
 Kamishima, T., Akaho, S.: Efficient clustering for orders. In: Zighed, D.A., Tsumoto, S., Ras, Z.W., Hacid, H. (eds.) Mining Complex Data, pp. 261–279. Springer, Berlin (2009)CrossRefGoogle Scholar
 Knuth, D.: The Art of Computer Programming, vol. 3. AddisonWesley, Boston (1998)zbMATHGoogle Scholar
 Kondor, R., Barbosa, M.: Ranking with kernels in Fourier space. In: Conference on Learning Theory (2010)Google Scholar
 Kondor, R., Howard, A., Jebara, T.: Multiobject tracking with representations of the symmetric group. In: AISTATS (2007)Google Scholar
 Lebanon, G., Mao, Y.: Nonparametric modeling of partially ranked data. J. Mach. Learn. Res. 9, 2401–2429 (2008)MathSciNetzbMATHGoogle Scholar
 Luce, R.D.: Individual Choice Behavior: A Theoretical Analysis. Wiley, New York (1959)zbMATHGoogle Scholar
 Mallows, C.L.: Nonnull ranking models. Biometrika 44, 114–130 (1957)MathSciNetCrossRefzbMATHGoogle Scholar
 Mania, H., Ramdas, A., Wainwright, M.J., Jordan, M.I., Recht, B.: On kernel methods for covariates that are rankings (2016)Google Scholar
 Mukherjee, S.: Estimation in exponential families. Ann. Stat. 44, 853–875 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
 Neal, R.M.: 1998. Regression and Classification Using Gaussian Process Priors. Bayesian Statistics 6Google Scholar
 Plackett, L.R.: The analysis of permutations. J. R. Stat. Soc. Ser. C (1974)Google Scholar
 Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)zbMATHGoogle Scholar
 Ross, S.M.: Simulation Fourth Edition, Statistical Modeling and Decision Science. Academic, Cambridge (2006)Google Scholar
 Schölkopf, B., Smola, A.: Learning with Kernels: Support Vector Machines, Regularisation, Optimisation and Beyond. MIT press, Cambridge (2002)Google Scholar
 Schölkopf, B., Smola, A.J., Müller, K.R.: Kernel principal component analysis. In: Schölkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods. MIT Press, Cambridge (1999)Google Scholar
 Sejdinovic, D., Sriperumbudur, B., Gretton, A., Fukumizu, K.: Equivalence of distancebased and RKHSbased statistics in hypothesis testing. Ann. Stat. 41, 2263–2291 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
 Serfling, R.J.: Approximation Theorems of Mathematical Statistics. Wiley, New York (1980)CrossRefzbMATHGoogle Scholar
 ShawerTaylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)CrossRefGoogle Scholar
 Sriperumbudur, B.K., Fukumizu, K., Lanckriet, G.R.G.: Universality, characteristic Kernels and RKHS embedding of measures. J. Mach. Learn. Res. 12, 2389–2410 (2011)MathSciNetzbMATHGoogle Scholar
 Stanley, R.P.: Enumerative Combinatorics, vol. 1. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
 Takeuchi, I., Le, Q.V., Sears, T.D., Smola, A.: Nonparametric quantile estimation. J. Mach. Learn. Res. 7, 1231–1264 (2006)MathSciNetzbMATHGoogle Scholar
 Tsuda, K., Kin, T., Asai, K.: Marginalised kernels for biological sequences. Bioinformatics 18, S268–S275 (2002)CrossRefGoogle Scholar
 Vitelli, V., Sørensen, Ø., Crispino, M., Frigessi, A., Arjas, E.: Probabilistic preference learning with the mallows rank model. J. Mach. Learn. Res. 18, 1581 (2017)MathSciNetzbMATHGoogle Scholar
 Wendland, H.: Scattered Data approximation. Cambridge University Press, Cambridge (2005)zbMATHGoogle Scholar
Copyright information
OpenAccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.