Advertisement

Information Retrieval Journal

, Volume 20, Issue 5, pp 480–505 | Cite as

The orchestration of a collaborative information seeking learning task

  • Simon Knight
  • Bart Rienties
  • Karen Littleton
  • Dirk Tempelaar
  • Matthew Mitsui
  • Chirag Shah
Search as Learning

Abstract

The paper describes our novel perspective on ‘searching to learn’ through collaborative information seeking (CIS). We describe this perspective, which motivated empirical work to ‘orchestrate’ a CIS searching to learn session. The work is described through the lens of orchestration, an approach which brings to the fore the ways in which: background context—including practical classroom constraints, and theoretical perspective; actors—including the educators, researchers, and technologies; and activities that are to be completed, are brought into alignment. The orchestration is exemplified through the description of research work designed to explore a pedagogically salient construct (epistemic cognition), in a particular institutional setting. Evaluation of the session indicated satisfaction with the orchestration from students, with written feedback indicating reflection from them on features of the orchestration. We foreground this approach to demonstrate the potential of orchestration as a design approach for researching and implementing CIS as a ‘searching to learn’ context.

Keywords

Collaborative information seeking Searching to learn Computer supported collaborative learning Educational technology 

Notes

Acknowledgements

The work reported in this paper was conducted while the first author was a Ph.D. candidate at the Open University, UK, to whom he extends thanks. Thanks too to Dr Dirk Tempelaar’s lab assistants at Maastricht University, who assisted in supporting the lab-based orchestration.

References

  1. Amershi, S., & Morris, M. R. (2008). CoSearch: A system for co-located collaborative web search. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, pp. 1647–1656. http://research.microsoft.com/en-us/um/people/merrie/papers/cosearch.pdf.
  2. Amershi, S., & Morris, M. R. (2009). Co-located collaborative web search, p. 3637. ACM Press. doi: 10.1145/1520340.1520547.
  3. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. doi: 10.1016/j.lindif.2013.01.007.CrossRefGoogle Scholar
  4. Ba, H., Tally, W., & Tsikalas, K. (2002). Investigating children’s emerging digital literacies. The Journal of Technology, Learning and Assessment, 1(4). http://www.edtechpolicy.org/ArchivedWebsites/JCTE/v1n4_jtla1.pdf.
  5. Barzilai, S., & Zohar, A. (2009). The Role of Epistemic Thinking in Online Learning. In Proceedings of the Chais Conference on Instructional Technologies Research 2009: Learning in the Technological Era. Raanana: The Open University of Israel. http://telem-pub.openu.ac.il/users/chais/2009/morning/1_2.pdf. Accessed 27 January 2012.
  6. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30(1), 39–85. doi: 10.1080/07370008.2011.636495.CrossRefGoogle Scholar
  7. Bateman, S., Gutwin, C. A., & McCalla, G. I. (2013). Social navigation for loosely-coupled information seeking in tightly-knit groups using webwear. In Proceedings of the 2013 Conference on Computer supported Cooperative Work, pp. 955–966. http://dl.acm.org/citation.cfm?id=2441885. Accessed 12 May 2013.
  8. Bloom, B. S. (1956). Taxonomy of educational objectives, handbook 1: Cognitive domain (2nd ed.). Boston: Addison Wesley Publishing Company.Google Scholar
  9. Böhm, T., Klas, C.-P., & Hemmje, M. (2016). Towards a probabilistic model for supporting collaborative information access. Information Retrieval Journal, 19(5), 487–509. doi: 10.1007/s10791-016-9285-3.CrossRefGoogle Scholar
  10. Bråten, I., & Weinstein, C. E. (2004). Internet-specific epistemological questionnaire (ISEQ). Austin, TX: Department of Educational Psychology, University of Texas at Austin.Google Scholar
  11. Bråten, I., & Strømsø, H. I. (2006). Epistemological beliefs, interest, and gender as predictors of internet-based learning activities. Computers in Human Behavior, 22(6), 1027–1042. doi: 10.1016/j.chb.2004.03.026.CrossRefGoogle Scholar
  12. Bråten, I., & Samuelstuen, M. S. (2007). Measuring strategic processing: Comparing task-specific self-reports to traces. Metacognition and Learning, 2(1), 1–20. doi: 10.1007/s11409-007-9004-y.CrossRefGoogle Scholar
  13. Bråten, I., & Strømsø, H. I. (2009). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes, 47(1), 1–31. doi: 10.1080/01638530902959646.CrossRefGoogle Scholar
  14. Bråten, I., Strømsø, H. I., & Samuelstuen, M. S. (2005). The relationship between internet-specific epistemological beliefs and learning within internet technologies. Journal of Educational Computing Research, 33(2), 141–171. doi: 10.2190/E763-X0LN-6NMF-CB86.CrossRefGoogle Scholar
  15. Bråten, I., Braasch, J. L. G., Strømsø, H. I., & Ferguson, L. E. (2014). Establishing trustworthiness when students read multiple documents containing conflicting scientific evidence. Reading Psychology. doi: 10.1080/02702711.2013.864362.Google Scholar
  16. Bromme, R., Pieschl, S., & Stahl, E. (2009). Epistemological beliefs are standards for adaptive learning: A functional theory about epistemological beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi: 10.1007/s11409-009-9053-5.CrossRefGoogle Scholar
  17. Chiu, Y.-L., Liang, J.-C., & Tsai, C.-C. (2013). Internet-specific epistemic beliefs and self-regulated learning in online academic information searching. Metacognition and Learning, 8(3), 235–260. doi: 10.1007/s11409-013-9103-x.CrossRefGoogle Scholar
  18. Dimopoulos, K., & Asimakopoulos, A. (2010). Science on the web: Secondary school students’ navigation patterns and preferred pages’ characteristics. Journal of Science Education and Technology, 19(3), 246–265. doi: 10.1007/s10956-009-9197-8.CrossRefGoogle Scholar
  19. Ferguson, R. (2012). The state of learning analytics in 2012: A review and future challenges (Technical Report No. kmi-12-01). Knowledge Media Institute: The Open University, UK. http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf. Accessed 22 October 2012.
  20. Ferguson, L. E., Bråten, I., Strømsø, H. I., & Anmarkrud, Ø. (2013). Epistemic beliefs and comprehension in the context of reading multiple documents: Examining the role of conflict. International Journal of Educational Research, 62, 100–114. doi: 10.1016/j.ijer.2013.07.001.CrossRefGoogle Scholar
  21. Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. doi: 10.1002/RRQ.027.Google Scholar
  22. Golovchinsky, G., & Diriye, A. (2011). Session-based search with Querium. In Proceedings of the HCIR 2011 Workshop, Mountain View, CA, USA. http://fxpal.com/publications/FXPAL-PR-11-660.pdf. Accessed 12 May 2013.
  23. Golovchinsky, G., Adcock, J., Pickens, J., Qvarfordt, P., & Back, M. (2008). Cerchiamo: A collaborative exploratory search tool. In Proceedings of Computer Supported Cooperative Work (CSCW). http://fxpal.fxpal.com/publications/FXPAL-PR-08-470.pdf. Accessed 12 May 2013.
  24. Greene, J. A., Muis, K. R., & Pieschl, S. (2010). The Role of epistemic beliefs in students’ self-regulated learning with computer-based learning environments: Conceptual and methodological issues. Educational Psychologist, 45(4), 245–257. doi: 10.1080/00461520.2010.515932.CrossRefGoogle Scholar
  25. Gubbels, M., Rose, A., Russell, D., & Bederson, B. B. (2012). SearchParty: Real-time Support for Social Learning in Synchronous Environments (No. HCIL-2012-21). Human-Computer Interaction Lab, University of Maryland. http://hcil2.cs.umd.edu/trs/2012-21/2012-21.pdf. Accessed 19 October 2012.
  26. Heilman, J. M., Kemmann, E., Bonert, M., Chatterjee, A., Ragar, B., Beards, G. M., et al. (2011). Wikipedia: A key tool for global public health promotion. Journal of medical Internet research, 13(1). http://www.ncbi.nlm.nih.gov/pmc/articles/pmc3221335/. Accessed 6 April 2015.
  27. Hertzum, M. (2008). Collaborative information seeking: The combined activity of information seeking and collaborative grounding. Information Processing and Management, 44(2), 957–962. doi: 10.1016/j.ipm.2007.03.007.CrossRefGoogle Scholar
  28. Hofer, B. K. (2001). Personal epistemology research: Implications for learning and teaching. Educational Psychology Review, 13(4), 353–383. http://www.springerlink.com/index/R6R6241GP87047K6.pdf. Accessed 24 October 2012.
  29. Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist, 39(1), 43–55. doi: 10.1207/s15326985ep3901_5.CrossRefGoogle Scholar
  30. Horgan, A., & Sweeney, J. (2012). University students’ online habits and their use of the internet for health information. CIN: Computers Informatics, Nursing, 30(8), 402–408. doi: 10.1097/NXN.0b013e3182510703.Google Scholar
  31. Hsu, L. (2014). An epistemological analysis of the application of an online inquiry-based program in tourism education. Australasian Journal of Educational Technology. doi: 10.14742/ajet.v30i1.402.Google Scholar
  32. Hsu, C.-Y., Tsai, M.-J., Hou, H.-T., & Tsai, C.-C. (2013). Epistemic beliefs, online search strategies, and behavioral patterns while exploring socioscientific issues. Journal of Science Education and Technology. doi: 10.1007/s10956-013-9477-1.Google Scholar
  33. Hwang, G.-J., Tsai, P.-S., Tsai, C.-C., & Tseng, J. C. R. (2008). A novel approach for assisting teachers in analyzing student web-searching behaviors. Computers & Education, 51(2), 926–938. doi: 10.1016/j.compedu.2007.09.011.CrossRefGoogle Scholar
  34. Hyldegård, J. (2006). Collaborative information behaviour—Exploring Kuhlthau’s Information Search Process model in a group-based educational setting. Information Processing and Management, 42(1), 276–298. doi: 10.1016/j.ipm.2004.06.013.CrossRefGoogle Scholar
  35. Hyldegård, J. (2009). Beyond the search process—Exploring group members’ information behavior in context. Information Processing and Management, 45(1), 142–158. doi: 10.1016/j.ipm.2008.05.007.CrossRefGoogle Scholar
  36. Knight, S. (2012). Finding knowledge—The role of dialogue in collaborative information retrieval in the classroom (master’s). Cambridge: University of Cambridge.Google Scholar
  37. Knight, S. (2015). Learning Indicators in SCIS Tasks. In 1st edition of the international workshop on the Evaluation on Collaborative Information Seeking and Retrieval ECol 2015. Presented at the 24th ACM International Conference on Information and Knowledge Management (CIKM 2015), Melbourne, Australia: ACM. doi:http://dx.doi.org/10.1145/2812376.2812378.
  38. Knight, S. (2016). Developing Learning Analytics for Epistemic Commitments in a Collaborative Information Seeking Environment (Ph.D.). Open University, Milton Keynes, UK. Retrieved from http://oro.open.ac.uk/46143/.
  39. Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., & Littleton, K. (2014). Epistemic networks for epistemic commitments. In International Conference of the Learning Sciences. Boulder, CO: International Society of the Learning Sciences. http://oro.open.ac.uk/39254/. Accessed 11 November 2013.
  40. Knight, S., & Littleton, K. (2015). Thinking, Interthinking, and technological tools. In R. Wegerif, L. Li, & J. C. Kaufman (Eds.), The routledge international handbook of research on teaching thinking (p. section 7(al)). Routledge. http://www.routledge.com/books/details/9780415747493/.
  41. Knight, S., & Littleton, K. (2016). Learning through Collaborative Information Seeking. In P. Hansen, C. Shah, & C.-P. Klas (Eds.), Collaborative information seeking: Best practices, new domains, new thoughts. Berlin: Springer.Google Scholar
  42. Knight, S., & Mercer, N. (2015). The role of exploratory talk in classroom search engine tasks. Technology, Pedagogy and Education, 24(3), 303–319. doi: 10.1080/1475939X.2014.931884.CrossRefGoogle Scholar
  43. Knight, S., & Mercer, N. (2016). The role of collaborative, epistemic discourse in classroom information seeking tasks. Technology, Pedagogy and Education, 26(1), 33–50. doi: 10.1080/1475939X.2016.1159978.CrossRefGoogle Scholar
  44. Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D. T., & Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the internet. Computers in Human Behavior, 73, 507–518. doi: 10.1016/j.chb.2017.04.014.CrossRefGoogle Scholar
  45. Kuiper, E., Volman, M., & Terwel, J. (2009). Developing Web literacy in collaborative inquiry activities. Computers & Education, 52(3), 668–680. doi: 10.1016/j.compedu.2008.11.010.CrossRefGoogle Scholar
  46. Lazonder, A. W. (2005). Do two heads search better than one? Effects of student collaboration on web search behaviour and search outcomes. British Journal of Educational Technology, 36(3), 465–475. doi: 10.1111/j.1467-8535.2005.00478.x.CrossRefGoogle Scholar
  47. Liechti, O., & Sumi, Y. (2002). Editorial: Awareness and the WWW. International Journal of Human-Computer Studies, 56(1), 1–5. doi: 10.1006/ijhc.2001.0512.CrossRefGoogle Scholar
  48. Lin, C., & Tsai, C. (2008). Exploring the structural relationships between high school students’ scientific epistemological views and their utilization of information commitments toward online science information. International Journal of Science Education, 30(15), 2001–2022. doi: 10.1080/09500690701613733.CrossRefGoogle Scholar
  49. Livingstone, S., Bober, M., & Helsper, E. (2005). Internet literacy among children and young people: Findings from the UK Children Go Online project. London: LSE Research Online. http://eprints.lse.ac.uk/397/1/UKCGOonlineLiteracy.pdf. Accessed 10 October 2011.
  50. Marchionini, G. (2006). Exploratory search: From finding to understanding. Communications of the ACM, 49(4), 41. doi: 10.1145/1121949.1121979.CrossRefGoogle Scholar
  51. Mason, L., Boldrin, A., & Ariasi, N. (2010). Epistemic metacognition in context: Evaluating and learning online information. Metacognition and Learning, 5(1), 67–90. doi: 10.1007/s11409-009-9048-2.CrossRefGoogle Scholar
  52. Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. doi: 10.1016/j.learninstruc.2010.01.001.CrossRefGoogle Scholar
  53. Moraveji, N., Morris, M. R., Morris, D., Czerwinski, M., & Henry Riche, N. (2011). ClassSearch: Facilitating the development of web search skills through social learning. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, pp. 1797–1806.Google Scholar
  54. Morris, M. R., & Horvitz, E. (2007). SearchTogether: An interface for collaborative web search. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 3–12. http://dl.acm.org/citation.cfm?id=1294215. Accessed 29 April 2013.
  55. Mozilla. (2014). Firefox. https://www.mozilla.org/en-US/firefox/new/. Accessed 6 April 2015.
  56. Newman, K., Knight, S., Elbeshausen, S., & Hansen, P. (2016). Situating CIS—The importance of context in collaborative information seeking. In P. Hansen, C. Shah, & C.-P. Klas (Eds.), Collaborative information seeking: Best practices, new domains, new thoughts. Berlin: Springer.Google Scholar
  57. OECD. (2013). PISA 2015: Draft reading literacy framework. OECD Publishing. http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Reading%20Framework%20.pdf. Accessed 22 April 2014.
  58. Paul, S., A., & Morris, M. R. (2009). CoSense: Enhancing sensemaking for collaborative web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1771–1780). New York, NY, USA: ACM. doi: 10.1145/1518701.1518974.
  59. Prieto, L. P., Dimitriadis, Y., Asensio-Pérez, J. I., & Looi, C.-K. (2015). Orchestration in learning technology research: Evaluation of a conceptual framework. Research in Learning Technology. doi: 10.3402/rlt.v23.28019.Google Scholar
  60. Prieto, L. P., Holenko Dlab, M., Gutiérrez, I., Abdulwahed, M., & Balid, W. (2011). Orchestrating technology enhanced learning: A literature review and a conceptual framework. International Journal of Technology Enhanced Learning, 3(6), 583–598. http://www.inderscienceonline.com/doi/abs/10.1504/IJTEL.2011.045449. Accessed 8 August 2016.
  61. Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning (1st ed.). Mahwah, NJ: Routledge.Google Scholar
  62. Schmidt, C. W. (2012). Trending now: Using social media to predict and track disease outbreaks. Environmental Health Perspectives, 120(1), a30–a33. doi: 10.1289/ehp.120-a30.CrossRefGoogle Scholar
  63. Schmidt, H. G., Molen, H. T. V. D., Winkel, W. W. R. T., & Wijnen, W. H. F. W. (2009). Constructivist, problem-based learning does work: A meta-analysis of curricular comparisons involving a single medical school. Educational Psychologist, 44(4), 227–249. doi: 10.1080/00461520903213592.CrossRefGoogle Scholar
  64. Schraw, G. (2001). Current themes and future directions in epistemological research: A commentary. Educational Psychology Review, 13(4), 451–464. doi: 10.1023/A:1011922015665.CrossRefGoogle Scholar
  65. Schraw, G. (2013). Conceptual integration and measurement of epistemological and ontological beliefs in educational research. ISRN Education. http://www.hindawi.com/isrn/education/aip/327680/. Accessed 15 January 2013.
  66. Shah, C. (2010). Coagmento-a collaborative information seeking, synthesis and sense-making framework. Integrated demo at CSCW, 6–11. http://research.microsoft.com/en-us/um/redmond/groups/connect/CSCW_10/docs/p527.pdf. Accessed 12 May 2013.
  67. Shah, C. (2012a). Information retrieval: Collaborative information seeking: The art and science of making the whole greater than the sum of all. Berlin: Springer.Google Scholar
  68. Shah, C. (2012b). Coagmento: A case study in designing. In E. Currás & N. L. Romero (Eds.), Systems science and collaborative information systems: Theories, practices and new research: Theories, practices and new research (p. 242). Hershey, PA: IGI Global. http://www.igi-global.com/chapter/coagmento-case-study-designing-user/61295. Accessed 13 June 2015.
  69. Shah, C. (2013). Effects of awareness on coordination in collaborative information seeking. Journal of the American Society for Information Science and Technology, n/a–n/a. doi: 10.1002/asi.22819.
  70. Shah, C. (2014). Coagmento. http://coagmento.org/. Accessed 6 April 2015.
  71. Shah, C., & González-Ibáñez, R. (2010). Exploring information seeking processes in collaborative search tasks. Proceedings of the American Society for Information Science and Technology, 47(1), 1–7. doi: 10.1002/meet.14504701211.CrossRefGoogle Scholar
  72. Shah, C., & Marchionini, G. (2010). Awareness in collaborative information seeking. Journal of the American Society for Information Science and Technology, 61(10), 1970–1986. doi: 10.1002/asi.21379.CrossRefGoogle Scholar
  73. Shah, C., Marchionini, G., & Kelly, D. (2009). Learning design principles for a collaborative information seeking system. In CHI’09 Extended Abstracts on Human Factors in Computing Systems (pp. 3419–3424). ACM. http://dl.acm.org/citation.cfm?id=1520496. Accessed 20 April 2015.
  74. Singh, R., Hsu, Y.-W., & Moon, N. (2013). Multiple perspective interactive search: A paradigm for exploratory search and information retrieval on the web. Multimedia Tools and Applications, 62(2), 507–543. http://link.springer.com/article/10.1007/s11042-011-0910-2. Accessed 27 August 2013.
  75. SQW. (2011). Evaluation of the home access programme—Final report. DCSF. https://www.education.gov.uk/publications/eOrderingDownload/DFE-RR132.pdf. Accessed 14 October 2011.
  76. Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’ trustworthiness? Educational Psychology, 31(2), 177–206. doi: 10.1080/01443410.2010.538039.CrossRefGoogle Scholar
  77. Sundin, O., & Johannisson, J. (2005). The instrumentality of information needs and relevance. In F. Crestani & I. Ruthven (Eds.), Context: Nature, impact, and role (Vol. 3507, pp. 107–118). Berlin, Heidelberg: Springer Berlin Heidelberg. http://www.springerlink.com/content/gb5gfmg8xn8u3gtx/. Accessed 31 January 2012.
  78. Tempelaar, D. T., Niculescu, A., Rienties, B., Gijselaers, W. H., & Giesbers, B. (2012). How achievement emotions impact students’ decisions for online learning, and what precedes those emotions. The Internet and Higher Education, 15(3), 161–169. doi: 10.1016/j.iheduc.2011.10.003.CrossRefGoogle Scholar
  79. Tsai, M.-J., Hsu, C.-Y., & Tsai, C.-C. (2012). Investigation of high school students’ online science information searching performance: The role of implicit and explicit strategies. Journal of Science Education and Technology, 21(2), 246–254. doi: 10.1007/s10956-011-9307-2.CrossRefGoogle Scholar
  80. Tsai, P.-S., Tsai, C.-C., & Hwang, G.-J. (2011). The correlates of Taiwan teachers’ epistemological beliefs concerning Internet environments, online search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54–63. doi: 10.1016/j.iheduc.2010.03.003.CrossRefGoogle Scholar
  81. Tseng, J. C. R., Hwang, G.-J., Tsai, P.-S., & Tsai, C.-C. (2009). Meta-analyzer: A web-based learning environment for analyzing student information searching behaviors. Computing, Information and Control, 5(3), 567–579. http://www.ijicic.org/icic07-is03-12-1.pdf. Accessed 18 September 2014.
  82. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional solutions. Computers in Human Behavior, 24(3), 623–648. doi: 10.1016/j.chb.2007.01.030.CrossRefGoogle Scholar
  83. Wildemuth, B. M., & Freund, L. (2012). Assigning search tasks designed to elicit exploratory search behaviors. In Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval (pp. 4:1–4:10). New York, NY, USA: ACM. doi: 10.1145/2391224.2391228.
  84. Yue, Z., Han, S., & He, D. (2012). Search tactics in collaborative exploratory web search. In Proceedings of HCIR 2012. Cambridge, MA. http://crystal.exp.sis.pitt.edu:8080/ips/paper/hcir2012_cs.pdf. Accessed 29 April 2013.

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Connected Intelligence CentreUniversity of Technology SydneySydneyAustralia
  2. 2.Institute for Educational TechnologyOpen UniversityMilton KeynesUK
  3. 3.Centre for Research in Education and Educational TechnologyOpen UniversityMilton KeynesUK
  4. 4.School of Business and EconomicsMaastricht UniversityMaastrichtNetherlands
  5. 5.School of Communication and InformationRutgers UniversityNew BrunswickUSA

Personalised recommendations