Probabilistic clustering of time-evolving distance data
- 934 Downloads
- 1 Citations
Abstract
We present a novel probabilistic clustering model for objects that are represented via pairwise distances and observed at different time points. The proposed method utilizes the information given by adjacent time points to find the underlying cluster structure and obtain a smooth cluster evolution. This approach allows the number of objects and clusters to differ at every time point, and no identification on the identities of the objects is needed. Further, the model does not require the number of clusters being specified in advance—they are instead determined automatically using a Dirichlet process prior. We validate our model on synthetic data showing that the proposed method is more accurate than state-of-the-art clustering methods. Finally, we use our dynamic clustering model to analyze and illustrate the evolution of brain cancer patients over time.
1 Introduction
A major challenge in data analysis is to find simple representations of the data that best reveal the underlying structure of the investigated phenomenon (Lee and Sebastian Seung 1999). Clustering is a powerful tool to detect such structure in empirical data, thus making it accessible to practitioners (Jain and Dubes 1988). The problem of clustering has a very long history in the data mining and machine learning communities, and numerous clustering algorithms and applications have been studied in many different scientific disciplines over the past 50 years (Jain 2008). Applications of clustering include a large variety of problem domains as, for example, clustering text, social networks, images, or biomedical data (Bandyopadhyay and Coyle 2003; Eisen et al. 1998; Ng et al. 2002; Steinbach et al. 2000). Traditional clustering methods such as k-means or Gaussian mixture models (Ferguson 1973), rely on geometric representation of the data. Nowadays, however, increasingly often there is no access to an underlying vectorial representation of the data since only pairwise similarities or distances are measured. An example application domain where such a setting frequently occurs is biomedical data analysis, where more often than not only pairwise distance data is available, e.g., when DNA or protein sequences are represented as pairwise distances or string alignments (Cuturi and Vert 2004; Leslie et al. 2003; Rätsch and Sonnenburg 2004; Saigo et al. 2004; Sonnenburg et al. 2007).
Although many clustering methods exist that work on distance data, including single linkage clustering, complete linkage clustering, and Ward’s clustering (Jain and Dubes 1988), these methods are static methods that are innocuous with respect to a potentially underlying time structure. However, when data is obtained at different points in time, dynamic models are needed that take a time component into account. For example in cancer research, genes are frequently measured at different time points, in order to examine the efficiency of a medication over time. In Network Security, HTTP connections are recorded at various timestamps, since network behaviors can quickly change over time; in Computer Vision, video streams contain time-indexed sequence of images. To deal with such scenarios, dynamic models that take the evolving nature of data into account are needed. Such a requirement has been addressed with evolutionary or dynamic clustering models for vectorial data [as for instance in Ahmed and Xing (2008), Blei and Frazier (2011), Teh et al. (2011), or Zhu et al. (2005)], which obtain a smooth clustering over multiple time points. However, to the best of our knowledge, no time-evolving clustering models exist that work on distance data directly, and clustering of time-evolving distance data is still an unsolved problem.
In this work we will bridge this gap and present a novel Bayesian time-evolving clustering model based on distance data directly that is specially tailored to temporal data and does not require direct access to an underlying vector space. Our model will be able to detect cluster popularity over time, based on the rich gets richer phenomenon. We will be able to make predictions about how popular a cluster will be at time \(t + 1\) if we already knew that it was a rich cluster at time point t. The assumption that rich clusters get richer seems plausible in many domains, for instance, a hot news topic is likely to stay hot for a given time period. Our model is also able to cope with variable data size: the number of data points may vary between time points, for instance, data items may arrive or leave. Also, the number of clusters may vary over time and the model is able to adjust its capacity accordingly, and automatically. The aim is to find the underlying structure at every time point and to obtain a smooth cluster evolution which results in an easily interpretable model. Thereby the information shared across neighboring time points is related to the size of the clusters, the time-varying property of the clusters is assumed to be Markovian, and Markov Chain Monte Carlo (MCMC) sampling is used for inference.
The presented method is also applicable for the less general case of pairwise similarity data, by using a slightly altered likelihood. Since Mercer kernels can encode similarities between many different kinds of objects (for instance kernels on graphs, images, structures or strings) the method proposed here can cover the entire scope of applications of kernel-based learning, be it string alignment kernels over DNA or protein sequences (Leslie et al. 2003; Rätsch and Sonnenburg 2004; Sonnenburg et al. 2007) or diffusion kernels on graphs (Vishwanathan et al. 2010).
We validate our approach by comparing it to baseline methods on simulated data where our new model significantly outperforms state-of-the-art clustering approaches. We apply our novel model to a highly topical and challenging real world data set of brain cancer patients from Memorial Sloan Kettering Cancer Center (MSKCC). This data consists of clinical notes as part of electronic health records (EHR) of brain cancer patients over 3 consecutive years. We model brain cancer patients over time where patients are grouped together based on the similarity of sentences in the clinical notes (see Sect. 4.2). All experiments were run on a 2.9 GHz Intel Core i5 processor with 8 GB RAM 1600 MHz, single core.
2 Background
In this section we recap important background knowledge which is essential for the remainder of this paper.
By moving from vectors X to pairwise similarities K and from similarities to pairwise distances D, there is a lack of information about geometric transformations: assume we only observe K without access to the vectorial representations \(X_{n\times d}\). Then we have lost the information about orthogonal transformations \(X\leftarrow X O\) with \(OO^t = I_d\), i.e. about rotations and reflections of the rows in X. If we only observe D, we have additionally lost the information about translations of the rows \(X \leftarrow X + (\varvec{1}_n \varvec{v}^t + \varvec{v}\varvec{1}_n^t) , \; \varvec{v} \in \mathbb {R}^d\).
The models above imply that the means in each row are expected to converge to zero as the number of replications d goes to infinity. Thus, if we had access to X and if we are not sure that the above zero-mean assumption holds, it might be a plausible strategy to subtract the empirical row means, \(X_{n\times d} \leftarrow X_{n\times d}- (1/d) X_{n\times d}\varvec{1}_d \varvec{1}_d^t\), and then to construct a candidate matrix K by computing the pairwise dot products. This procedure should be statistically robust if \(d\gg n\), since then the empirical means are probably close to their expected values. Such a corrected matrix K fulfills two important requirements for selecting candidate dot product matrices:
First, K should be “typical” with respect to the assumed Wishart model with \(\varvec{\mu }= \varvec{0}\), thereby avoiding any bias introduced by a particular choice. Second, the choice should be robust in a statistical sense: if we are given a second observation from the same underlying data source, the two selected prototypical matrices \(K_1\) and \(K_2\) should be similar. For small d, this correction procedure is dangerous since it can introduce a strong bias even if the model is correct: suppose we are given two replications from \(N(\varvec{\mu }= \varvec{0}_n, \varSigma = \varSigma _B)\), i.e. \(d=2\). After subtracting the row means, all row vectors lie on the diagonal line in \(\mathbb {R}^2\), and the cluster structure is heavily distorted.
Consider now the case where we observe K without access to X. For “correcting” the matrix K just as described above we would need a procedure which effectively subtracts the empirical row means from the rows of X.
Unfortunately, there exists no such matrix transformation that operates directly on K without explicit construction of X. It is important to note that the “usual” centering transformation \(K \leftarrow QK Q\) with \(Q_{ij} = \delta _{ij}-\frac{1}{n}\) as used in kernel PCA and related algorithms does not work here: in kernel PCA the rows of X are assumed to be i.i.d. replications in \(\mathbb {R}^d\). Consequently, the centered matrix \(K_c\) is built by subtracting the column means: \(X_{n\times d} \leftarrow X_{n\times d} - (1/n) \varvec{1}_n \varvec{1}_n^t X_{n\times d}\) and \( K_c = XX^t = QKQ\). Here, we need to subtract the row means, and therefore it is necessary to explicitly construct X, which implies that we have to choose a certain orthogonal transformation O. It might be reasonable to consider only rotations and to use the principal components as coordinate axes. This is essentially the kernel PCA embedding procedure: compute \(K_c = QKQ\) and its eigenvalue decomposition \(K_c = V \varLambda V^t\), and then project on the principal axes: \(X = V \varLambda ^{1/2}\). The problem with this vector-space embedding is that it is statistically robust in the above sense only if d is small, because otherwise the directions of the principal axes might be difficult to estimate, and the estimates for two replicated observations might highly fluctuate, leading to different column-mean normalizations. Note that this condition for fixing the rotation contradicts the above condition \(d\gg n\) that justifies the subtraction of the means. Further, column mean normalization will change the pairwise dissimilarities \(D_{ij}\) (even if the model is correct!), and this change can be drastic if d is small.
3 A time-evolving translation-invariant Wishart–Dirichlet process
In this section, we present a novel dynamic clustering approach, the time-evolving translation-invariant Wishart–Dirichlet process (Te-TiWD) for clustering distance data that is available at multiple time points. In this model, we assume that pairwise distance data \(D_t\) with \( 1\le t \le T\) is available over T time points. At every time point t all objects are fully exchangeable, and the number of data points may differ at the different time points. This model clusters data points over multiple time points, allowing group memberships and the number of clusters to evolve over time by addition, deletion or change in existing clusters. The model is based on the static clustering model that was proposed in Vogt et al. (2010) which is not able to account for a time structure.
Notations used throughout this manuscript
\(D_t\) | Distance matrix at time point t [cf. (6)] |
\(\varDelta _t\) | \(\varDelta \) matrix at time point t [cf. (7)] |
\(B_t\) | Partition matrix at time point t |
\(k_{b_t}\) | Number of blocks \(b_t\) present in the partition \(B_t\) |
\(n_{b_t}\) | The size of block \(b_t\) |
\(n_{b_t}^{(-l)}\) | Size of block \(b_t\) without object l |
\(n_t\) | Number of data points present at the t-th time point |
\(A_t\) | \(k_{b_t}\times k_{b_t}\) matrix |
\(A_{t_{ij}}\) | The between-class variance of block i and block j |
\([B_t]_{t=1}^T\) | is defined as \((B_1, B_2,\ldots ,B_T)\) |
\([A_t]_{t=1}^T\) | is defined as \((A_1, A_2,\ldots ,A_T)\) |
\(p([B_t]_{t=1}^T)\) | \(p(B_1)p(B_2|B_1)\ldots p(B_T|B_{T-1})\) defines a first-order Markov chain |
\(p([A_t]_{t=1}^T)\) | \( p(A_1)p(A_2|A_1)\ldots p(A_T|A_{T-1})\) defines a first-order Markov chain |
\([B]_{t-}\) | B matrices at all time points except at time point t |
3.1 The model
The aim of the proposed method is to cluster distance data \(D_t\) at multiple time points, for \( 1\le t \le T\). For every time point under consideration, t, we obtain a distance matrix \(D_t\) and we want to infer the partition matrix \(B_t\), by utilizing the partitions from adjacent time points. By using information from adjacent time points, we expect better clustering results than clustering every time point independently. At every time point, the number of data points may differ, and some clusters may die out or evolve over time. The assumptions on the data are the following:
Assumption 1
Note that \(\varSigma _{A_t}\) is a more general version of \(\beta B\):
Lemma 1
\(\varSigma _{A_t}=\beta B\) iff \(A_{t_{ij}}=\left\{ \begin{array}{cc} 0&{}\quad \, \, if \, \, i \ne j\\ \beta \, \, &{}\quad if \, \, i=j \end{array} \right. . \)
Prior over\(A_t\). The prior over the \(A_t\) matrices is given by a Wishart distribution, \(P(A_t|A_{t-1}) \sim \mathcal {W}_d(A_t|A_{t-1})\) and \(S_0:=P(A_{1})= \mathcal {W}_d(A_{1}|I_{k_{b_1}})\). The degrees of freedom d influences the behavior of the Wishart distribution: a low value for d allows drastic changes in the clustering structure, a high value for d allows fewer changes. We also have to consider that the size of \(A_{t-1}\), \(A_{t}\) and \(A_{t+1}\) might differ, as it is possible that the number of clusters in every epoch is different. Therefore, we consider the following two cases:
1. if there are more blocks at time \(t-1\)than at time t, i. e. \(k_{b_{t-1}}>k_{b_t}\):
delete corresponding rows and columns in \(A_{t-1}\). With \(A'_{t-1}\) we denote the “reduced” matrix. Then it holds that \(A_{t} \sim \mathcal {W}_d(A'_{t-1})\)
2. if there are fewer blocks at time \(t-1\)than at time t, i. e. if \(k_{b_{t-1}}<k_{b_t}\):
3.1.1 MCMC sampling for posterior inference
Table of prior probabilities
\({c_{\mathrm{new}}}\) exists at | \( \hbox {P}(\hbox {l}=\hbox {c}_\mathrm{new}| \hbox {B}_\mathrm{t-1})\hbox {P}(\hbox {B}_\mathrm{t+1}|\hbox {B}_\mathrm{t})\) |
---|---|
Both time points \(t-1\) and \(t+1\) | \(\propto (n_{c_{t-1}} \cdot \frac{\xi }{m})\cdot n_{c_{t+1}}\qquad (19)\) |
Time point \(t-1\) but not at time point \(t+1\) | \( \propto \frac{\xi }{m} \cdot n_{c_{t-1}}\qquad (20)\) |
Time point \(t+1\) but not at time point \(t-1\) | \(\propto \frac{\xi }{m} \cdot n_{c_{t+1}}\qquad (21)\) |
Neither \(t-1\) nor \(t+1\), (i.e. l belongs to a completely new cluster) | \(\propto \frac{\xi }{m} \qquad (22)\) |
Metropolis–Hastings update steps In every time point, we need to sample \(\beta \) values in the between-class variance matrix \(\varSigma _{A_t}\). To find the \(\beta \) values within one epoch, we sample the whole “new” \(A_t\) matrix, denoted by \(A_{t_{new}}\), with a Metropolis–Hastings algorithm (see Robert and Casella 2005). With \(A_{t_{old}}\) we denote the initial \(A_t\) matrix. As proposal distribution we chose a Wishart distribution, leading to \(P(A_{t_{new}}|A_{t_{old}})\sim \mathcal {W}(A_{t_{new}}|A_{t_{old}})\) and \(P(A_{t_{new}})\sim \mathcal {W}(A_{t_{new}}|I_{k_{b_t}}).\)
Hyperparameters and initialization Our model includes the following hyperparameters: the scale parameter \(\alpha \), the number k of clusters, the Dirichlet rate \(\xi \), the degrees of freedom d and a scale parameter s. The model is not sensitive to the choice of s, and we fix s to 1. \(\alpha \) is sampled from a Gamma distribution with shape and scale parameters r and m. For the number k of clusters, our framework is applicable to two scenarios: we can either assume \(k=\infty \) which results in the CRP model, or we fix k to a large constant which can be viewed as a truncated Ewens process. As the model does not suffer from the label switching problem, initialization is not a crucial problem. We initialize the block size with size 1, i. e. we start with one cluster for all objects. The Dirichlet rate \(\xi \) only weakly influences the likelihood, and the variance only decays with \(1/\log (n_t)\) (see Ewens 1972). In practice, we should not expect to reliably estimate \(\xi \). Rather, we should have some intuition about \(\xi \), maybe guided by the observation that under the Ewens process model the probability of two objects belonging to the same cluster is \(1/(1+\xi )\). We can then either define an appropriate prior distribution, or we can fix \(\xi \). Due to the weak effect of \(\xi \) on conditionals, these approaches are usually very similar. The degrees of freedom d can be estimated by the rank of K, if it is known from a pre-processing procedure. As d is not a very critical parameter (all likelihood contributions are basically raised to the power of d), d might also be used as an annealing-type parameter for freezing a representative partition in the limit for \(d\rightarrow \infty \).
Complexity We define one sweep of the Gibbs sampler as one complete update of \((B_t, A_t)\). The most time consuming part in a sweep is the update of \(B_t\) by re-estimating the assignments to blocks for a single object (characterized by a row/column in \(D_t\)), given the partition of the remaining objects. Therefore we have to compute the membership probabilities in all existing blocks (and in a new block). Every time a new partition is analyzed, a naive implementation requires \(O(n^3)\) costs for computing the determinant of \(\tilde{W}_t\) and the product \(\tilde{W}_tD_t\). In one sweep we need to compute \(k_{b_t}\) such probabilities for \(n_t\) objects, summing up to costs of \(O(n^4k_{b_t})\). This suggests that the scalability to large datasets can pose a problem. In this regard we plan to address run time in future work by investigating the potential of variational methods, parallelizing the MCMC sampler and by updating parameters associated with multiple time points simultaneously.
Identifiability of clusters In some applications, it is of interest to identify and track clusters over time. For example by grouping newspaper articles into topics it might be interesting to know which topics are present over a long time period, when a new topic becomes popular and when a former popular topic dies out. Due to the translation-invariance of our novel longitudinal model, we additionally need a cluster mean to be able to track clusters over the time course. To estimate the mean of the clusters we propose to embed the “overall” data matrix \(D^*\in \mathbb {R}^{N \times N}\) with \(N:=\sum _{t=1}^T n_t\) that contains the pairwise distances between all objects over all time points into a vector space, using kernel PCA. We first construct a positive semi-definite matrix \(K^*\) which fulfills \(D^*_{ij}=K^*_{ii}+K^*_{jj}-2K^*_{ij}\). For correcting \(K^*\), we compute the centered matrix \(K^*_c=Q^*K^*Q^*\) with \(Q^*_{ij}=\delta _{ij}-\frac{1}{N}\). As a next step, we compute the eigenvalue decomposition of \(K^*_c\), i.e. \(K^*_c=V\varLambda V^T\) and then project on the principal axes \(X^*=V\varLambda ^{\frac{1}{2}}\), i.e. we use the principal components as coordinate axes. By embedding the distances \(D^*\) into a vector space, the underlying block structure might be distorted (see Fig. 2). As our aim is to find the underlying block structure, it is hence infeasible to embed the data for clustering. But, for tracking the clusters, we just need to find the mean of an already inferred block structure, i.e. we embed the data not for grouping data points, but for finding a mean of an already assigned partition that allows us to track the clusters over time. We embed all objects together and choose the same orthogonal transformations for all objects, which enables identifiability of cluster means over the time course. This preprocessing step is only necessary if one is interested in the identifiability of clusters, and \(X^*\) needs only to be computed once outside the sampling routine. Since computing \(X^*\) is computationally expensive, it is done only once as a preprocessing step if required. Computing \(X^*\) within the sampling routine would slow down our sampler significantly.
4 Experiments
4.1 Synthetic experiments
4.1.1 Well separated clusters
In a first experiment, we test our method on simulated data. We simulate data in two ways, first we generate data points accordingly to the model assumptions, and secondly we generate data independent of the model assumptions. We start with a small experiment where we consider five time points each with 20 data points per time point in 100 dimensions, i. e. we consider a small data set size and large dimension problem.
- (a)500 Gibbs sweeps are computed for the Te-TiWD cluster process (after a burn-in phase of 250 sweeps). We check convergence of the algorithm by analyzing the trace plot of the number of blocks \(k_{b_t}\) during sampling. A trace plot is a plot of the iteration number against the value of the draw of the parameter at each iteration, in our case the number of blocks \(k_{b_t}\). On a trace plot one can visually see whether a chain gets stuck in certain areas of the parameter space, which indicates bad mixing, and one can also observe after how many sweeps the sampler stabilizes (the number of sweeps depends on the size of the data set). We observe a remarkable stability of the sampler (compared to the usual situations in traditional mixture models), which follows from the fact that no label-switching can appear. Finally, we perform an annealing procedure to freeze a certain partition. Here, d is used as an annealing-type parameter for freezing a representative partition in the limit \(d\rightarrow \infty \). On our machine, this experiment took roughly 4 min, and the sampler stabilizes after roughly 50 sweeps. As the ground truth is known, we can compute the adjusted rand index as an indicator for the accuracy of the Te-TiWD model. We repeat the clustering process 50 times. The result is shown in form of a box plot (Te-TIWD) in Fig. 6.
- (b)
In order to compare the performance of the time-evolving model (Te-TiWD) to baseline models, we also run the static probabilistic clustering process as well as hierarchical clustering models (Ward, complete linkage and single linkage) on every time point separately and compute the averaged accuracy over all time points. For the comparison to the static probabilistic method (Vogt et al. 2010), we use the same set-up as for Te-TiWD, we run 500 Gibbs sweeps with a burn-in phase of 250 sweeps and repeat it for 50 times. For the hierarchical methods, the resulting trees are cut at the number of clusters found by the nonparametric probabilistic model. Accuracy is computed for every time point separately, and then averaged over all time points. In this scenario, the static clustering models performs almost as well as the time-evolving clustering, see Fig. 6, as expected in such a setting where all groups are well separated at every single time point.
- (c)
As a further comparison to a baseline dynamic clustering model, we embed the distances into a Euclidean vector space and run a Gaussian dynamic clustering model (Te-Gauss) on the embedded vectorial data. As the clusters are well separated, embedding the data and clustering on vectors works well, as shown in box plot “Te-Gauss” in Fig. 6.
- (d)
As a last comparison we evaluate a pooled clustering over all time points. For this experiment, we not only need the pairwise distances at every single time point, but also the pairwise distances of objects across all time points. The number of sweeps and repetitions remains the same as in the experiments above. We conduct one clustering over all objects of all time points, and after clustering, we extract the objects belonging to the same time point and compute the rand index on every time point separately. This experiment shows worse results (see box plot “pooled” in Fig. 6), which can be explained as follows: by combining all time points to one data matrix, new clusters over all time points are found, this means clusters are shifted and objects over time are grouped together, introducing new clusters by reforming boundaries of old clusters. These new clusters inhibit objects to group together which would group together at single time points, destroying the underlying “true” cluster structure.
4.1.2 Highly overlapping clusters
4.1.3 Data generation independent of model assumptions.
4.2 Analysis of brain cancer patient based on electronic health records (EHR)
We apply our proposed model to a dataset of clinical notes from brain cancer patients at Memorial Sloan Kettering Cancer Center (MSKCC). Brain cancer patients make up 1.4 % of all cancer patients, annually. Survival is highly variable, depending on age, gender, cancer subtype, and progression when diagnosed, but on average 33 % of patients survive the first 5 years. As a first step, we partition a total of 195,297 sentences from 3,403 electronic health records (EHR) from 704 MSKCC brain cancer patients into groups of similar vocabulary. This is done by treating sentences as binary vectors with non-zero entries corresponding to vocabulary, and obtaining a similarity measure using ranked neighborhood comparisons (Vogt 2015). Sentences are clustered using this similarity measure with the Louvain method (Blondel et al. 2008). The sentence clusters do not employ any form of negation detection, and so interpreting them can be a little tricky. We use the context of the sentence cluster’s topic as well as any additional information to help interpret the meaning of a cluster. Using these sentence clusters as features, we obtain patient similarities with the same ranked neighborhood comparison method. We partition the patients documents into windows of 1 year each, and obtain three time points where enough documents are available to compute similarities between patients. At each year, we represent a patient with a binary vector whose length is the number of sentence clusters. A non-zero entry corresponds to an occurrence of that sentence cluster in the patient’s corpus during the specified time period.
In the first year, we have 704 patients, in the second year 170 and in the third year 123 patients. This data set has specific features which make our model particularly suitable. First, the number of patients differs in every year. Second, patients disappear over the time course, either due to death or due to leaving the hospital. Third, patients do not necessarily need to have a document every year, so a patient can be absent from year 2 and appear in year 3. This gap occurred a total of 31 times in our data set.
This is why our flexible model is very well suited for this problem, as the model can deal with changing numbers of objects and changing number of clusters in every year, clusters can disappear or reappear, as well as patients. The result of our clustering model is shown in Fig. 10. On our machine, this experiment took roughly 6 h, and the sampler stabilizes after roughly 500 sweeps.
Modeling patients over time provides important insights for automated analyses and medical doctors, as it is possible to check for every patient how the state of the patient as represented by the cluster membership changes over time. Also, if a new patient enters the study, one can infer, based on similarity to other patients, how to classify and possibly treat this patient best or to suggest clinical trials for each patient. Such clustering methods therefore make an important step towards solving the technical challenges of personalized cancer treatment.
5 Conclusion
In this work, we propose a novel dynamic Bayesian clustering model to cluster time-evolving distance data. A probabilistic model that is able to handle non-vectorial data in form of pairwise distances has the advantage that there is no need to embed the data into a vector space. To summarize, our contributions in this work are five-fold: (i) We develop a dynamic probabilistic clustering approach that circumvents the potentially problematic data embedding step by directly operating on pairwise time-evolving distance data. (ii) Our model enables to track the clusters over time, giving information about clusters that die out or emerge over time. (iii) By using a Dirichlet process prior, there is no need to fix the number of clusters in advance. (iv) We test and validate our model on simulated data. We compare the performance of our new method with baseline probabilistic and hierarchical clustering methods. (v) We use our model to cluster brain cancer patients into similar subgroups over a time course of 3 years. Dynamic partitioning of patients would play an important role in cancer treatment, as it enables inference from groups of similar patients to an individual. Such an inference can help medical doctors to adapt or optimize existing treatments, assign billing codes, or predict survival times for a patient based on similar patients in the same group.
Notes
Acknowledgments
We thank Natalie Davidson, Theofanis Karaletsos and David Kuo for helpful discussions and suggestions. JV and MK were partly funded through postdoctoral fellowships awarded by the Swiss National Science Foundation (SNSF; under PBBSP2_146758) and by the German Research Foundation (DFG; under Kl 2698/1-1 and VO 2003/1-1), respectively. MK acknowledges support by the German Research Foundation through the grant KL 2698/2-1. We gratefully acknowledge funding from Memorial Sloan Kettering Cancer Center and the National Cancer Institute (Grant 1R01CA176785-01A1). Access to patient data is covered under IRB Waiver #WA0426-13.
References
- Adametz, D., & Roth, V. (2011). Bayesian partitioning of large-scale distance data. In NIPS, pp. 1368–1376.Google Scholar
- Ahmed, A., & Xing, E. (2008). Dynamic non-parametric mixture models and the recurrent Chinese restaurant process: with applications to evolutionary clustering. In Proceedings of the eighth SIAM international conference on data mining (SDM).Google Scholar
- Anderson, T. W. (1946). The non-central wishart distribution and certain problems of multivariate statistics. The Annals of Mathematical Statistics, 17(4), 409–431.CrossRefMATHGoogle Scholar
- Bandyopadhyay, S, & Coyle, E. J. (2003). An energy efficient hierarchical clustering algorithm for wireless sensor networks. In INFOCOM 2003. twenty-second annual joint conference of the IEEE computer and communications (Vol. 3, pp. 1713–1723). IEEE Societies.Google Scholar
- Bilodeau, M., & Brenner, D. (1999). Theory of multivariate statistics. Berlin: Springer.Google Scholar
- Blei, D., & Jordan, M. (2006). Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1, 121–144.MathSciNetCrossRefGoogle Scholar
- Blei, D. M., & Frazier, P. (2011). Distance dependent chinese restaurant processes. Journal of Machine Learning Reseach, 12(12), 2461–2488.MathSciNetGoogle Scholar
- Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008, P10008.Google Scholar
- Cuturi, M., & Vert, J.-P. (2004). A mutual information kernel for strings. In Proceedings of the international joint conference on neural network.Google Scholar
- Eisen, M. B., Spellman, P. T., Brown, P. O., & Botstein, D. (1998). Cluster analysis and display of genome-wide expression patterns. Proceedings of the National Academy of Sciences, 95(25), 14863–14868.CrossRefGoogle Scholar
- Ewens, W. J. (1972). The sampling theory of selectively neutral alleles. Theoretical Population Biology, 3, 87–112.MathSciNetCrossRefMATHGoogle Scholar
- Ferguson, T. S. (1973). A bayesian analysis of some nonparametric problems. Annals of Statistics, 1, 209–230.MathSciNetCrossRefMATHGoogle Scholar
- Jain, A. K. (2008). Data clustering: 50 years beyond k-means. Pattern Recognition Letters, 31(8), 651–666.Google Scholar
- Jain, A. K., & Dubes, R. C. (1988). Algorithms for clustering data. Upper Saddle River: Prentice-Hall Inc.MATHGoogle Scholar
- Lee, D. D., & Sebastian Seung, H. (1999). Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755), 788–791.CrossRefMATHGoogle Scholar
- Leslie, C., Eskin, E., Cohen, A., Weston, J., & Noble, W. S. (2003). Mismatch string kernel for discriminative protein classification. Bioinformatics, 1(1), 1–10.Google Scholar
- MacEachern, S. N. (1994). Estimating normal means with a conjugate-style Dirichlet process prior. Communication in Statistics: Simulation and Computation, 23, 727–741.MathSciNetCrossRefMATHGoogle Scholar
- McCullagh, P. (2009). Marginal likelihood for distance matrices. Statistica Sinica, 19, 631–649.MathSciNetMATHGoogle Scholar
- McCullagh, P., & Yang, J. (2008). How many clusters? Bayesian Analysis, 3, 101–120.MathSciNetCrossRefMATHGoogle Scholar
- Neal, R. M. (2000). Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9, 249–265.MathSciNetGoogle Scholar
- Ng, A. Y., Jordan, M. I., Weiss, Y., et al. (2002). On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems, 2, 849–856.Google Scholar
- Pitman, J. (2006). Combinatorial stochastic processes. In J. Picard (Ed.), Ecole d’Ete de Probabilites de Saint-Flour XXXII-2002. Berlin: Springer.Google Scholar
- Rätsch, G., & Sonnenburg, S. (2004). Accurate splice site prediction for caenorhabditis elegans. In Kernel methods in computational biology, MIT Press series on computational molecular biology (pp. 277–298). Cambridge: MIT Press.Google Scholar
- Robert, C. P., & Casella, G. (2005). Monte Carlo statistical methods. Berlin: Springer.Google Scholar
- Saigo, H., Vert, J.-P., Ueda, N., & Akutsu, T. (2004). Protein homology detection using string alignment kernels. Bioinformatics, 20(11), 1682–1689.CrossRefGoogle Scholar
- Sonnenburg, S., Rätsch, G., & Rieck, K. (2007). Large scale learning with string kernels. In L. Bottou, O. Chapelle, D. DeCoste, & J. Weston (Eds.), Large Scale Kernel Machines (pp. 73–103). Cambridge, MA: MIT Press.Google Scholar
- Srivastava, M. S. (2003). Singular Wishart and multivariate beta distributions. Annals of Statistics, 31(2), 1537–1560.MathSciNetCrossRefMATHGoogle Scholar
- Steinbach, M., Karypis, G., Kumar, V. et al. (2000). A comparison of document clustering techniques. In KDD workshop on text mining (Vol. 400, pp. 525–526). Boston.Google Scholar
- Teh, Y. W., Blundell, C., & Elliott, L. T. (2011). Modelling genetic variations with fragmentation-coagulation processes. In Advances in neural information processing systems.Google Scholar
- Vishwanathan, S. V. N., Schraudolph, N. N., Kondor, R., & Borgwardt, K. M. (2010). Graph kernels. The Journal of Machine Learning Research, 11, 1201–1242.MathSciNetMATHGoogle Scholar
- Vogt, J. E. (2015). IEEE/ACM Transactions on Computational Biology and Bioinformatics. Unsupervised structure detection in biomedical data.Google Scholar
- Vogt, J. E., Prabhakaran, S., Fuchs, T. J., & Roth, V. (2010). The translation-invariant Wishart–Dirichlet process for clustering distance data. In ICML, pp. 1111–1118.Google Scholar
- Zhu, X., Ghahramani, Z., & Lafferty, J. (2005). Time-sensitive dirichlet process mixture models. Technical report, Carnegie Mellon University.Google Scholar