# Deep collective matrix factorization for augmented multi-view learning

- 26 Downloads

**Part of the following topical collections:**

## Abstract

Learning by integrating multiple heterogeneous data sources is a common requirement in many tasks. Collective Matrix Factorization (CMF) is a technique to learn shared latent representations from arbitrary collections of matrices. It can be used to simultaneously complete one or more matrices, for predicting the unknown entries. Classical CMF methods assume linearity in the interaction of latent factors which can be restrictive and fails to capture complex non-linear interactions. In this paper, we develop the first deep-learning based method, called dCMF, for unsupervised learning of multiple shared representations, that can model such non-linear interactions, from an arbitrary collection of matrices. We address optimization challenges that arise due to dependencies between shared representations through multi-task Bayesian optimization and design an acquisition function adapted for collective learning of hyperparameters. Our experiments show that dCMF significantly outperforms previous CMF algorithms in integrating heterogeneous data for predictive modeling. Further, on two tasks—recommendation and prediction of gene-disease association—dCMF outperforms state-of-the-art matrix completion algorithms that can utilize auxiliary sources of information.

## Keywords

Collective Matrix Factorization Deep learning Augmented multi-view learning Bayesian optimization Recommendation Gene-disease prioritization## Notes

## References

- Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013). Deep canonical correlation analysis. In
*Proceedings of the 30th international conference on machine learning*, pp. 1247–1255.Google Scholar - Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*,*35*(8), 1798–1828.Google Scholar - Bergstra, J. S., Bardenet, R., Bengio, Y., & Kégl, B. (2011). Algorithms for hyper-parameter optimization. In
*Proceedings of the 24th international conference on neural information processing systems*, pp. 2546–2554.Google Scholar - Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization.
*Journal of Machine Learning Research*,*13*, 281–305.MathSciNetzbMATHGoogle Scholar - Bonilla, E. V., Chai, K. M., & Williams, C. (2007). Multi-task Gaussian process prediction. In
*Proceedings of the 20th international conference on neural information processing systems*, pp. 153–160.Google Scholar - Bouchard, G., Yin, D., & Guo, S. (2013). Convex collective matrix factorization. In
*Proceedings of the sixteenth international conference on artificial intelligence and statistics*, pp. 144–152.Google Scholar - Boutros, M., & Ahringer, J. (2008). The art and design of genetic screens: RNA interference.
*Nature Reviews Genetics*,*9*(7), 554.Google Scholar - Chang, S., Han, W., Tang, J., Qi, G.-J., Aggarwal, C. C., & Huang, T. S. (2015). Heterogeneous network embedding via deep architectures. In
*Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining*, pp. 119–128. ACM.Google Scholar - Chen, M., Xu, Z., Weinberger, K., & Sha, F. (2012). Marginalized denoising autoencoders for domain adaptation. In
*Proceedings of the 29th international conference on machine learning*, pp. 1627–1634.Google Scholar - Coburn, T. C. (2000). Geostatistics for natural resources evaluation.
*Technometrics*,*42*(4), 437–438.Google Scholar - Cui, P., Wang, X., Pei, J., & Zhu, W. (2019). A survey on network embedding.
*IEEE Transactions on Knowledge and Data Engineering*,*31*(5), 833–852.Google Scholar - Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets.
*Journal of Machine Learning Research*,*7*, 1–30.MathSciNetzbMATHGoogle Scholar - Dong, X., Yu, L., Wu, Z., Sun, Y., Yuan, L., & Zhang, F. (2017). A hybrid collaborative filtering model with deep structure for recommender systems. In
*Proceedings of the thirty-first AAAI conference on artificial intelligence*, pp. 1309–1315.Google Scholar - Frayling, T. M. (2007). Genome-wide association studies provide new insights into type 2 diabetes aetiology.
*Nature Reviews Genetics*,*8*(9), 657.Google Scholar - Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016).
*Deep learning*(Vol. 1). Cambridge: MIT Press.zbMATHGoogle Scholar - Guo, X., Gao, L., Liu, X., & Yin, J. (2017). Improved deep embedded clustering with local structure preservation. In
*Proceedings of the twenty-sixth international joint conference on artificial intelligence*, pp. 1753–1759.Google Scholar - Han, X., Shi, C., Wang, S., Philip, S. Y., & Song, L. (2018). Aspect-level deep collaborative filtering via heterogeneous information networks. In
*Proceedings of the twenty-seventh international joint conference on artificial intelligence*, pp. 3393–3399.Google Scholar - Hardoon, D. R., Szedmak, S., & Shawe-Taylor, J. (2004). Canonical correlation analysis: An overview with application to learning methods.
*Neural Computation*,*16*(12), 2639–2664.zbMATHGoogle Scholar - Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks.
*Science*,*313*(5786), 504–507.MathSciNetzbMATHGoogle Scholar - Hotelling, H. (1936). Relations between two sets of variates.
*Biometrika*,*28*(3/4), 321–377.zbMATHGoogle Scholar - Hu, Y., Zhang, D., Ye, J., Li, X., & He, X. (2013). Fast and accurate matrix completion via truncated nuclear norm regularization.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*,*35*(9), 2117–2130.Google Scholar - Jones, D. R. (2001). A taxonomy of global optimization methods based on response surfaces.
*Journal of Global Optimization*,*21*(4), 345–383.MathSciNetzbMATHGoogle Scholar - Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. In
*International conference on learning representations*.Google Scholar - Klami, A., Bouchard, G., & Tripathi, A. (2014). Group-sparse embeddings in collective matrix factorization. In
*International conference on learning representations*.Google Scholar - Knowles, J. (2006). ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems.
*IEEE Transactions on Evolutionary Computation*,*10*(1), 50–66.Google Scholar - Kolker, E., et al. (2015). Finding text-supported gene-to-disease co-appearances with MOPED-Digger.
*Omics: A Journal of Integrative Biology*,*19*(12), 754–756.Google Scholar - Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems.
*Computer*,*42*(8), 30–37.Google Scholar - Lan, C., Wang, J., & Huan, J. (2016). Towards a theoretical understanding of negative transfer in collective matrix factorization. In
*Proceedings of the thirty-second conference on uncertainty in artificial intelligence*, pp. 367–376.Google Scholar - Lee, I., Blom, U. M., Wang, P. I., Shim, J. E., & Marcotte, E. M. (2011). Prioritizing candidate disease genes by network-based boosting of genome-wide association data.
*Genome Research*,*21*(7), 1109–1121.Google Scholar - Li, X., & She, J. (2017). Collaborative variational autoencoder for recommender systems. In
*Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining*, pp. 305–314.Google Scholar - Li, S., Kawale, J., & Fu, Y. (2015). Deep collaborative filtering via marginalized denoising auto-encoder. In
*Proceedings of the 24th ACM international conference on information and knowledge management*, pp. 811–820.Google Scholar - Liu, J., Wang, D., & Ding, Y. (2017). PHD: A probabilistic model of hybrid deep collaborative filtering for recommender systems. In
*Proceedings of the ninth Asian conference on machine learning*, pp. 224–239.Google Scholar - Loguercio, S., Good, B. M., & Su, A. I. (2013). Dizeez: An online game for human gene-disease annotation.
*PLoS ONE*,*8*(8), 71171.Google Scholar - Natarajan, N., & Dhillon, I. S. (2014). Inductive matrix completion for predicting gene-disease associations.
*Bioinformatics*,*30*(12), 60–68.Google Scholar - Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., & Ng, A. Y. (2011). Multimodal deep learning. In
*Proceedings of the 28th international conference on machine learning*, pp. 689–696.Google Scholar - Opap, K., & Mulder, N. (2017). Recent advances in predicting gene-disease associations.
*F1000Research*6.Google Scholar - Pazzani, M., & Billsus, D. (1997). Learning and revising user profiles: The identification of interesting web sites.
*Machine Learning*,*27*(3), 313–331.Google Scholar - Pers, T. H., et al. (2011). Meta-analysis of heterogeneous data sources for genome-scale identification of risk genes in complex phenotypes.
*Genetic Epidemiology*,*35*(5), 318–332.Google Scholar - Piñero, J., et al. (2016). DisGeNET: A comprehensive platform integrating information on human disease-associated genes and variants.
*Nucleic Acids Research*,*45*(D1), 833–839.Google Scholar - Piro, R. M., & Di Cunto, F. (2012). Computational approaches to disease-gene prediction: Rationale, classification and successes.
*The FEBS Journal*,*279*(5), 678–696.Google Scholar - Schuyler, P. L., Hole, W. T., Tuttle, M. S., & Sherertz, D. D. (1993). The UMLS metathesaurus: Representing different views of biomedical concepts.
*Bulletin of the Medical Library Association*,*81*(2), 217.Google Scholar - Seyyedrazzagi, E., & Navimipour, N. J. (2017). Disease genes prioritizing mechanisms: A comprehensive and systematic literature review.
*Network Modeling Analysis in Health Informatics and Bioinformatics*,*6*(1), 13.Google Scholar - Shi, C., Hu, B., Zhao, W. X., & Philip, S. Y. (2019). Heterogeneous information network embedding for recommendation.
*IEEE Transactions on Knowledge and Data Engineering*,*31*(2), 357–370.Google Scholar - Shi, C., & Philip, S. Y. (2017).
*Heterogeneous information network analysis and applications*. Berlin: Springer.Google Scholar - Singh, A. P., & Gordon, G. J. (2008). Relational learning via collective matrix factorization. In
*Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining*, pp. 650–658.Google Scholar - Singh-Blom, U. M., et al. (2013). Prediction and validation of gene-disease associations using methods inspired by social network analyses.
*PLoS ONE*,*8*(5), 58977.Google Scholar - Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. In
*Proceedings of the 25th international conference on neural information processing systems*, pp. 2951–2959.Google Scholar - Srebro, N., & Shraibman, A. (2005). Rank, trace-norm and max-norm. In
*International conference on computational learning theory*, pp. 545–560.Google Scholar - Swersky, K., Snoek, J., & Adams, R. P. (2013). Multi-task Bayesian optimization. In
*Proceedings of the 26th international conference on neural information processing systems*, pp. 2004–2012.Google Scholar - Vincent, P., et al. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.
*Journal of Machine Learning Research*,*11*, 3371–3408.MathSciNetzbMATHGoogle Scholar - Wang, W., Arora, R., Livescu, K., & Bilmes, J. (2015). On Deep Multi-view Representation Learning. In
*Proceedings of the 32nd international conference on machine learning*, pp. 1083–1092.Google Scholar - Wang, Q., Sun, M., Zhan, L., Thompson, P., Ji, S., & Zhou, J. (2017). Multi-modality disease modeling via collective deep matrix factorization. In
*Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining*, pp. 1155–1164.Google Scholar - Wang, H., Wang, N., & Yeung, D.-Y. (2015). Collaborative deep learning for recommender systems. In
*Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining*, pp. 1235–1244.Google Scholar - Weinstein, J. N., et al. (2013). The Cancer Genome Atlas Pan-Cancer analysis project.
*Nature Genetics*,*45*(10), 1113–1120.Google Scholar - Zeng, X., Liao, Y., Liu, Y., & Zou, Q. (2017). Prediction and validation of disease genes using HeteSim scores.
*IEEE/ACM Transactions on Computational Biology and Bioinformatics*,*14*(3), 687–695.Google Scholar - Zhou, H., & Skolnick, J. (2016). A knowledge-based approach for predicting gene-disease associations.
*Bioinformatics*,*32*(18), 2831–2838.Google Scholar