Data Mining in Data-Intensive and Cognitively-Complex Settings: Lessons Learned from the Dicode Project

  • Natalja Friesen
  • Jörg Kindermann
  • Doris Maassen
  • Stefan Rüping
Part of the Studies in Big Data book series (SBD, volume 5)


This chapter reports on practical lessons learned while developing the Dicode’s data mining services and using them in data-intensive and cognitively-complex settings. Various sources were taken into consideration to establish these lessons, including user feedbacks obtained from evaluation studies, discussion in teams, as well as observation of services’ usage. The lessons are presented in a way that could aid people who engage in various phases of developing similar kind of systems.


Data mining framework Data mining services Text mining services Big data Hadoop Storm Semantic technologies 


  1. 1.
    Marz, N., Warren, J.: Big Data—Principles and Best Practices of Scalable Real-Time Data Systems. Manning Publications, New York (2012)Google Scholar
  2. 2.
    Baron, P.: Big Data für IT-Entscheider. Riesige Datenmengen und moderne Technologien gewinnbringend nutzen, München (2013)CrossRefGoogle Scholar
  3. 3.
    Grosskreutz, H., Paurat D.: Fast and memory-efficient discovery of the top-k relevant subgroups in a reduced candidate space. In: Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Science vol. 6911, pp. 533–548. Springer, Heidelberg (2011)Google Scholar
  4. 4.
    Han, J., Pei, J., Yin, Y.: Mining frequent patterns without candidate generation. In: Proceedings of SIGMOD’00. pp. 1–12. ACM Press, New York (2000)
  5. 5.
    Büttcher, S., Clarke, C., Cormack, G.: Information Retrieval: Implementing and Evaluating Search Engines. MIT Press, Cambridge, Mass (2010)Google Scholar
  6. 6.
    Friesen, N., Rüping, S.: Distance Metric Learning for Recommender Systems in Complex Domains. In: Proceedings of dicoSyn 2012 (Mastering Data-Intensive Collaboration through the Synergy of Human and Machine Reasoning), a workshop at CSCW 2012, February 12, 2012, Seattle (2012)Google Scholar
  7. 7.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)MATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Natalja Friesen
    • 1
  • Jörg Kindermann
    • 1
  • Doris Maassen
    • 2
  • Stefan Rüping
    • 1
  1. 1.Fraunhofer IAISSankt AugustinGermany
  2. 2.Neofonie GmbHBerlinGermany

Personalised recommendations