Recommending Collaborative Filtering Algorithms Using Subsampling Landmarkers

  • Tiago CunhaEmail author
  • Carlos Soares
  • André C. P. L. F. de Carvalho
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10558)


Recommender Systems have become increasingly popular, propelling the emergence of several algorithms. As the number of algorithms grows, the selection of the most suitable algorithm for a new task becomes more complex. The development of new Recommender Systems would benefit from tools to support the selection of the most suitable algorithm. Metalearning has been used for similar purposes in other tasks, such as classification and regression. It learns predictive models to map characteristics of a dataset with the predictive performance obtained by a set of algorithms. For such, different types of characteristics have been proposed: statistical and/or information-theoretical, model-based and landmarkers. Recent studies argue that landmarkers are successful in selecting algorithms for different tasks. We propose a set of landmarkers for a Metalearning approach to the selection of Collaborative Filtering algorithms. The performance is compared with a state of the art systematic metafeatures approach using statistical and/or information-theoretical metafeatures. The results show that the metalevel accuracy performance using landmarkers is not statistically significantly better than the metafeatures obtained with a more traditional approach. Furthermore, the baselevel results obtained with the algorithms recommended using landmarkers are worse than the ones obtained with the other metafeatures. In summary, our results show that, contrary to the results obtained in other tasks, these landmarkers are not necessarily the best metafeatures for algorithm selection in Collaborative Filtering.


Metalearning Subsampling landmarkers Collaborative filtering 



This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 under the Portugal 2020 Partnership Agreement, and through the Portuguese National Innovation Agency (ANI) as a part of project «FASCOM | POCI-01-0247-FEDER-003506».


  1. 1.
    Adomavicius, G., Tuzhilin, A.: Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Trans. on Knowl. and Data Eng. 17(6), 734–749 (2005)CrossRefGoogle Scholar
  2. 2.
    Adomavicius, G., Zhang, J.: Impact of data characteristics on recommender systems performance. ACM Trans. Manag. Inf. Syst. 3(1), 1–17 (2012)CrossRefGoogle Scholar
  3. 3.
    Bensusan, H., Kalousis, A.: Estimating the Predictive Accuracy of a Classifier. In: European Conference on Machine Learning, pp. 25–36 (2001)Google Scholar
  4. 4.
    Bobadilla, J., Ortega, F., Hernando, A., Gutiérrez, A.: Recommender systems survey. Knowl.-Based Syst. 46, 109–132 (2013)CrossRefGoogle Scholar
  5. 5.
    Brazdil, P., Giraud-Carrier, C., Soares, C., Vilalta, R.: Metalearning: Applications to Data Mining, 1st edn. Springer, Heidelberg (2009). doi: 10.1007/978-3-540-73263-1 zbMATHGoogle Scholar
  6. 6.
    Brazdil, P., Soares, C., da Costa, J.: Ranking Learning Algorithms : Using IBL and Meta-Learning on Accuracy and Time Results. Mach. Learn. 50(3), 251–277 (2003)CrossRefzbMATHGoogle Scholar
  7. 7.
    Cunha, T., Soares, C., de Carvalho, A.C.: Selecting Collaborative Filtering algorithms using Metalearning. In: European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 393–409 (2016)Google Scholar
  8. 8.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Dooms, S., De Pessemier, T., Martens, L.: MovieTweetings: a Movie Rating Dataset Collected From Twitter. In: CrowdRec at RecSys 2013 (2013)Google Scholar
  10. 10.
    Ekstrand, M., Riedl, J.: When recommenders fail: predicting recommender failure for algorithm selection and combination. In: ACM Conference on Recommender Systems, pp. 233–236 (2012)Google Scholar
  11. 11.
    Fürnkranz, J., Petrak, J., Bradzil, P., Soares, C.: On the use of fast subsampling estimates for algorithm recommendation. Technical report (2002)Google Scholar
  12. 12.
    Gantner, Z., Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: MyMediaLite: a free recommender system library. In: ACM Conference on Recommender Systems, pp. 305–308 (2011)Google Scholar
  13. 13.
    Goldberg, K., Roeder, T., Gupta, D., Perkins, C.: Eigentaste: A Constant Time Collaborative Filtering Algorithm. Inf. Retr. 4(2), 133–151 (2001)CrossRefzbMATHGoogle Scholar
  14. 14.
    Griffith, J., O’Riordan, C., Sorensen, H.: Investigations into user rating information and predictive accuracy in a collaborative filtering domain. In: ACM Symposium on Applied Computing, pp. 937–942 (2012)Google Scholar
  15. 15.
    GroupLens: MovieLens datasets (2016).
  16. 16.
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)CrossRefGoogle Scholar
  17. 17.
    Kanda, J., de Carvalho, A., Hruschka, E., Soares, C., Brazdil, P.: Meta-learning to select the best meta-heuristic for the Traveling Salesman Problem: A comparison of meta-features. Neurocomputing 205, 393–406 (2016)CrossRefGoogle Scholar
  18. 18.
    Kück, M., Crone, S.F., Freitag, M.: Meta-learning with neural networks and landmarking for forecasting model selection - an empirical evaluation of different feature sets applied to industry data meta-learning with neural networks and landmarking for forecasting model selection. In: International Joint Conference on Neural Networks. pp. 1499–1506 (2016)Google Scholar
  19. 19.
    Kuhn, M.: caret: Classification and Regression Training (2016)., r package version 6.0-73
  20. 20.
    Ler, D., Koprinska, I., Chawla, S.: Utilizing regression-based landmarkers within a meta-learning framework for algorithm selection. School of Information Technologies University of Sydney, Technical report (2005)Google Scholar
  21. 21.
    Ler, D., Koprinska, I., Chawla, S.: Utilizing regression-based landmarkers within a meta-learning framework for algorithm selection. In: Proceedings of the ICML-2005 Workshop on Metalearning, pp. 44–51 (2005)Google Scholar
  22. 22.
    Lü, L., Medo, M., Yeung, C.H., Zhang, Y.C., Zhang, Z.K., Zhou, T.: Recommender systems. Phys. Rep. 519(1), 1–49 (2012)CrossRefGoogle Scholar
  23. 23.
    Matuszyk, P., Spiliopoulou, M.: Predicting the performance of collaborative filtering algorithms. In: International Conference on Web Intelligence, Mining and Semantics. pp. 38:1–38:6 (2014)Google Scholar
  24. 24.
    McAuley, J., Leskovec, J.: Hidden factors and hidden topics: understanding rating dimensions with review text. In: ACM Conference on Recommender Systems, pp. 165–172 (2013)Google Scholar
  25. 25.
    Pfahringer, B., Bensusan, H., Giraud-Carrier, C.: Meta-learning by landmarking various learning algorithms. In: International Conference on Machine Learning, pp. 743–750 (2000)Google Scholar
  26. 26.
    Pinto, F., Soares, C., Mendes-Moreira, J.: Towards automatic generation of Metafeatures. In: Pacific Asia Conference on Knowledge Discovery and Data Mining, pp. 215–226 (2016)Google Scholar
  27. 27.
    Prudêncio, R.B., Ludermir, T.B.: Meta-learning approaches to selecting time series models. Neurocomputing 61, 121–137 (2004)CrossRefGoogle Scholar
  28. 28.
    Rossi, A.L.D., de Carvalho, A.C.P.D.L.F., Soares, C., de Souza, B.F.: MetaStream: A meta-learning based method for periodic algorithm selection in time-changing data. Neurocomputing 127, 52–64 (2014)CrossRefGoogle Scholar
  29. 29.
    Serban, F., Vanschoren, J., Bernstein, A.: A survey of intelligent assistants for data analysis. ACM Comput. Surv. 47(212), 1–35 (2013)CrossRefGoogle Scholar
  30. 30.
    Vanschoren, J.: Understanding machine learning performance with experiment databases. Ph.D. thesis, Katholieke Universiteit Leuven (2010)Google Scholar
  31. 31.
    Wang, H., Lu, Y., Zhai, C.: Latent Aspect Rating Analysis Without Aspect Keyword Supervision. In: ACM SIGKDD, KDD 2011, pp. 618–626. ACM (2011)Google Scholar
  32. 32.
    Yahoo!: Webscope datasets (2016).
  33. 33.
    Yang, X., Guo, Y., Liu, Y., Steck, H.: A survey of collaborative filtering based social recommender systems. Comput. Commun. 41, 1–10 (2014)CrossRefGoogle Scholar
  34. 34.
    Yelp: Yelp dataset challenge (2016).
  35. 35.
    Zafarani, R., Liu, H.: Social computing data repository at ASU (2009).
  36. 36.
    Ziegler, C.N.C., McNee, S.M.S., Konstan, J.a.J., Lausen, G.: Improving recommendation lists through topic diversification. In: Proceedings of the 14th International Conference on World Wide Web WWW 2005, p. 22. ACM (2005)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Tiago Cunha
    • 1
    Email author
  • Carlos Soares
    • 1
  • André C. P. L. F. de Carvalho
    • 2
  1. 1.INESC-TEC/FEUPPortoPortugal
  2. 2.ICMC - USPSão CarlosBrazil

Personalised recommendations