1 Introduction

Computer-based Information Retrieval (IR) has been around for at least 40 years, and its origins can be traced back to the late 1940s if not earlier. The discipline has been transformed since the explosion of the Internet and the Web. Search has entered popular culture and it is now a hot topic discussed in mass media, with widespread interest in the subject shown by the public at large. Many academic papers are published each year showing advances in the field, and significant funds are invested in research and development projects in which ideas and technology are successfully transferred to industry and business. This has a direct effect on people’s lives, as they use IR tools either consciously or unconsciously for a very wide range of tasks, such as work, pleasure, etc.

Users of IR fall into two major categories that are non-mutually exclusive: those who develop and evaluate IR systems and services and those who consume them. The former are researchers and developers in disciplines such as computing and information sciences, while the latter are everyday users of the technology. Both of these groups have educational needs. Computer science students require knowledge of fundamental issues in IR so they can implement the technology and further extend knowledge in the area by developing new theories and models, an example being postgraduate students. IR can also be used to reinforce the knowledge acquired in other subjects including programming, algorithms and data structures, and user interface design. Library and information science students require knowledge of advanced search techniques, resolving information needs and evaluation methodologies, etc. Other students just use IR systems to resolve their own information needs, e.g. search for information to write an essay. When looking at the subject from the educational point of view, it is clear that ‘IR forms an ideal subject for exploring creative teaching, learning and assessment methods’ (Jones 2007).

The objective of this review is to provide an overview of the literature in the area of pedagogy and information retrieval—it is by no means exhaustive as much of the pedagogical literature can be applied to the subject of IR. The material presented here can be used in a number of ways. One obvious use is to give the reader an idea of how to go about teaching the subject, i.e. teaching and learning methods that may help in designing and organizing courses and thus improve student learning experience. Lecturers can therefore use the research presented here to gain knowledge of best practices of other colleagues, experiences or methodologies which they can use to improve their own teaching. The review can also be used by people who have a pedagogical research interest in teaching and learning, and need an overview of work already carried out. We outline experiences of people who reported a problem in teaching IR, thought about this problem and how to solve it, and found an appropriate solution. IR researchers should also find this paper useful. Although they may not be concerned directly with teaching and learning, they should be aware that output of their research could be presented in IR courses, disseminating methodologies, techniques, models, open problems, etc., which they have devised.

A taxonomy was developed and used in organizing this paper. The taxonomy which reflects on the main aspects of teaching and learning IR is shown in Fig. 1.

Fig. 1
figure 1

Organization of the review on teaching and learning IR

Full size image

The taxonomy has two levels of abstraction, both focused on different aspects of the same problems—namely pedagogy and IR. Level 1 focuses on the subject itself, and the different fields in which IR can be applied. The two main areas are Library and Information Science and Computer Science. The relationship between these two areas is not a dichotomous one and we relate the continuous nature of the technical level in IR to different aspects of the field (see Sect. 2). Each area has its own particular education goals given the requisite technical level, which is the subject of Sect. 3. Level 2 of the taxonomy focuses on the pedagogical aspect in IR, that is how methods from the world of education can be applied to the subject, in terms of teaching and learning methods (Sect. 4), assessment and feedback (Sect. 5) and finally curricula design (Sect. 6). There are overlaps in the taxonomy, but it does give us a good overall structure to present our work. The review follows the structure outlined by the taxonomy and discusses the work at each level and subdivision. This is followed by a brief citation survey of the literature used in the review, and the conclusions.

2 Technical levels (non-technical to highly technical)

Before discussing educational goals, it would be useful to outline the technical level of the proposed education, in order to establish precisely what is to be achieved. This can range very widely indeed, with non-technical subjects such as user needs for information being studied in LIS, while highly technical subjects such as compression for inverted lists being covered in CS courses, e.g. Cacheda et al. (2008), who utilize the programming skills of Master students to produce a web search engine. A definition of IR is useful in this context—the indexing of objects (text, images, video, etc.) for retrieval by a user. LIS students are more interested in indexing/classifying the objects using meta-data/thesauri, while CS students write the software to undertake the indexing.

The technical level is in our view not dichotomous, but is a continuum from non-technical to technical. It is possible that some technical issues are covered in LIS courses, e.g. the syntax of search systems such as Dialog. Johnson (2008) who teaches LIS students argues that knowledge of technical issues in search “helps students to understand the challenge of search in modern information environments”. It is also possible for some user issues to be covered on more technical courses e.g. understanding of user requirements when building user interfaces (UI’s) to IR systems. Ruthven et al. (2008) describe an assessment approach for designing for users using a holistic viewpoint including UI’s. Figure 2 illustrates this technical continuum with some examples of where disciplines may lie above the line and subjects that might be taught in any of the disciplines below the line.

Fig. 2
figure 2

Technical continuum: non-technical to highly technical

Full size image

The placement of subjects and examples on this continuum is very subjective and open to interpretation (not an unknown problem in IR!), but it does give a flavour of the technical needs of particular subjects. We do not infer that a discipline on one end of the scale would never be taught a particular subject, but there is less likelihood that it would happen e.g. Thornley (2008) describes a method of teaching IR as a philosophical problem aimed largely at LIS students, which could be of use to those studying CS. Education from both main areas can be used to inform each other and, as this review will show, there has always been some cross fertilization of ideas in IR education between LIS and CS to the benefit of all students. Teaching in other subjects including those detailed in Fig. 2 can also benefit from the interaction.

3 Educational goals

Information Retrieval has a long history in the Library and Information Sciences disciplines. Mooers (1951) introduced the term Information Retrieval for the first time in the context of Library documentation—this seminal paper provides a set of common objectives in LIS programs. Using these objectives the students can develop different knowledge and skills such as formulating questions based on a user’s information need, identifying potential sources of information, developing successful search strategies or evaluating the results of a search. However as technology developed, a need for education in computer science (CS) was required in order to support these activities and a new set of objectives for that field focusing on more technical issues was set (Atchison et al. 1968). Students in LIS and CS can benefit from these stated objectives, although at different ends of the technical scale (see Sect. 2 above). We focus on the more technical issues in Sect. 3.1 and move on to goals in LIS in Sect. 3.2.

3.1 Computer science (CS)

IR is not a core part of the curricula in CS (see Sect. 6) and as a consequence it is difficult to find courses in a CS degree devoted exclusively to this subject—although they are beginning to appear more frequently in many masters or doctoral programs. When looking at IR education in CS the focus is less on searching for information and more on the technical and implementation aspects, i.e. how information is stored in computer systems, retrieval models, efficiency issues, evaluation methods, and, of course, how to develop implicit techniques to facilitate search that are transparent to end users.

In order to tackle the educational goals, we need to define what might be expected in the IR field from a CS graduate. In general, we might expect that computer scientists devise new approaches for searching, develop the products and design complex, high-performance networks to cope with immense quantities of data (IEEE and ACM 2001). Therefore, a solid foundation in technical skills from both engineering and informatics areas are needed. Obviously, these broad competences cannot be covered by means of a unique IR course in the CS curricula, but they might be acquired through the whole degree instead. For example, IR can be used as an application field in a data structures course illustrating the advantages of using different data structured to store an index or, in the case of a network (web) based course, a project application using a web crawler. Our emphasis here is on the educational goals of an IR based course at the undergraduate level. We note that the particular goals in master’s level could well vary significantly between different universities, mainly because at this level the research component receives special attention and each research unit has its own particular expertise.

The learning objectives of the elective unit “Information storage and retrieval (IM11)” from the Information Management (IM) area (IEEE and ACM 2001) are:

  1. 1)

    Explain basic information storage and retrieval concepts.

  2. 2)

    Describe what issues are specific to efficient information retrieval.

  3. 3)

    Give applications of alternative search strategies and explain why the particular search strategy is appropriate for the application.

  4. 4)

    Perform Internet-based research.

  5. 5)

    Design and implement a small to medium size information storage and retrieval system.

Important criticisms can be made of these learning objectives—for example ‘perform Internet-based research’ is somewhat ambiguous and need not refer to IR per se. The lack of discussion about an important issue—evaluation—is particularly problematic. However, this is a generally agreed standard, and it is therefore useful to extract educational goals from in the specialized teaching and learning literature in order to establish a match of educational goals between the two. We focus on the questions by presenting educational goals found in the literature, given the known limitations of the IM11 objectives.

The goal of understanding fundamental aspects of IR is presented by several authors. Henrich and Morgenroth (2007) show basic concepts of IR in a first course; Efthimiadis and Hendry (2005) and Hendry and Efthimiadis (2008) support knowledge acquisition of basic technical concepts for search engines as an important kind of literacy and also to improve the students’ conceptual model of how search engines work. In Jones (2007), Zhu and Tang (2006), Argamon et al. (2005) and Brusilovsky (2002), the pedagogical objective is to give an overview of IR together with its components and an understanding of the interaction between these components. The objective of this educational goal is to inform the design of curricula (see Sect. 6 for more details). In other work, issues such as learning advanced techniques of IR are found as goals (Henrich and Morgenroth 2007; Herrera-Viedma et al. 2007; de Campos et al. 2007; Goharian et al. 2004).

With respect to search strategies, the main goal of instruction is to develop the learners’ practical capability to search and understand the heuristic nature of IR techniques (Airio et al. 2007; Halttunen 2007; Halttunen and Järvelin 2005). Another educational goal is to improve the search skills of the students (Halttunen 2003a; Brajnik et al. 2003; Fourie and van Brakel 1995). Jacobson and Ignacio (1997) in addition to improving search skills, set pedagogical goal of learning to retrieve information using a variety of systems, include their main features and how these can be applied to particular tasks.

Brandt and Uden (2003) offer a compromise between the two objectives stated directly above, as they teach basic issues in IR to give students’ background in the area, in order to further their knowledge of search skills.

With respect to IR systems in more advanced courses, the main objective expressed is to acquire skills to develop new IR methods using software modules already developed (Calado et al. 2007; de Campos et al. 2007; Jinguji et al. 2006; Chau et al. 2003). The goal of understanding of the whole IR process is also pursued by the lecturers. Taking this idea a step further, another goal is to train the students’ to think about relevant issues when analyzing problems and offer viable solutions using IR tools (Hendry 2007; Marshall et al. 2006; Meng 2003).

We have identified three main educational goals in the literature (we match each to the learning objective number from CS Curricula stated above):

  1. A.

    Knowledge on IR foundations (1 and 2)

  2. B.

    Training in search strategies (3).

  3. C.

    Knowledge on Information retrieval systems: processes and components (4 and 5).

The educational goals stated here correspond to the goals described in the Computer Science Curricula.

An IR course should therefore teach the main elements of the IR discipline to impart a general idea of the area; strategies for searching (given an information need, how to express modify and update a query depending on the results, analysis of the search engine output, use of different tools to help the formulation of the query, etc.); and topics related to the development of IRSs (an understanding of the different components, how they interact, how to implement them, how to add new retrieval models, evaluation modules, etc.).

In order to understand the potential impact of IR education, we analyzed a selection of job announcements in the field of IR concerned with computing, obtained from several mailing lists such as SIG-IRList, IR, Doceng, UM, UAI. From this analysis we have found the following requirements in most of the adverts:

  • Strong knowledge of the Information Retrieval (IR) field (A).

  • Deep familiarity and hands-on experience in IR techniques (B).

  • Ability to conduct experiments involving massive data sources (C).

  • Understand those features relevant to Search Engine Optimization (C).

Most of these requirements can be placed in at least one of the three educational goals that we have extracted from the literature or the five learning objectives from the CS Curricula (we place them in our identified education goals). This provides some evidence that instructors and IR course designers are attempting to fulfil the needs of the industry in the field of IR, as the educational goals we identify are included in most job requirements. The lack of explicit reference to evaluation here should be noted.

3.2 Library and information science (LIS)

As stated above, information retrieval has a long history in the field of LIS. The main reason for this is that the skills required to do information retrieval, i.e. searching, are core to the function of a search intermediary or reference librarian (Ingwersen 1992). A search intermediary’s role is to understand the information needs of a user from a given domain (e.g. law, medicine), match it to a relevant resource and write a query to be submitted to that source to resolve the users’ information need. Skills required including management aspects of information provision—what information should be provided, what tools are to be utilized, how are they evaluated, what access mechanism are used etc. This role of information professionals has evolved since the nineties and now it also involves teaching search skills to end users. The educational goal in LIS modules is therefore to provide the skills to the student that will allow them to serve in these multifaceted roles.

Professional and accrediting organisations have a clear role in defining educational goals this area. For example the Charted Institute of Library and Information Professionals (CILIP) in their core schema for the body of professional knowledge (CLILP 2004), specify that an understanding of the information need and user behaviour is required between the user/client and the information itself. The UK Quality Assurance Agency for Higher Education (QAA) lists a number of specific requisite skills and qualities in their subject benchmark for Bachelors degree with honours that are clearly relevant to the search intermediary role (QAA 2000):

  • “2.6 The ability to identify, analyse and evaluate the information needs of different groups and make informed decisions to satisfy them. Students should be aware of methods of obtaining feedback from users”,

  • “2.10 Information retrieval skills in the user of primary and secondary sources irrespective of medium”,

  • “2.11 The ability to create and use finding aids or retrieval tools and a knowledge and understanding of the techniques and standards for their creation”.

Bodies therefore define the required skills at different levels of abstraction from very abstract (CLILP 2004) to more specific (QAA 2000). This requisite knowledge must be delivered in some form (whatever the abstraction, although more abstract implies more flexibility) through modules delivered on LIS courses.

Bates et al. (2005), survey LIS curricula in Europe and identify three aspects areas of information seeking and retrieval (IS&R) which inform the educational goals of programmes on that continent. This gives us some idea of how educational goals set by professional and accrediting organisations are implemented in practice. Consider the following diagram in Fig. 3.

Fig. 3
figure 3

Aspects of information seeking and retrieval (IS&R)

Full size image

This is Wilson’s nested model (Wilson 1999), and it was used to specify the relationship between the three aspects of IS&R. Each aspect informs the other and according to the evidence provided by Bates et al. (2005) “neither of the two inner layers can be understood without some appreciation of the layer outside them”. Definitions of each of the layers in terms of educational goals are as follows (outmost layer first):

  • Human Information Behaviour (HIB): an understanding is required of how users create, interact and use information from various sources and communication channels.

  • Information Seeking: the process in HIB of looking for information to fulfil an information need.

  • Information Retrieval: use of information retrieval systems in information seeking, understanding the underlying technologies and evaluating the results of searches applied on those technologies.

Typical skills specified in each of the areas are as follows (this is not exhaustive; please refer to Bates et al. (2005) for a fuller list):

  • Human Information Behaviour: understanding of people, sources and places, patterns of behaviour, information needs.

  • Information Seeking: understanding of information seeking in context (e.g. professional, personal), strategies and tactics for IS, relevance and satisfaction.

  • Information Retrieval: understanding of search models (best match, exact match), the role of metadata and controlled vocabularies in retrieval, evaluation of systems and services.

Some of these skills clear match the skills specified in the QAA subject benchmark above, apart from some aspects of HIB. These are undoubtly covered in other aspects of the benchmark however. It should be noted that not all skills specified in modules surveyed in Europe necessary come from either professional or accrediting bodies, but it is noteworthy that there are no contradictions in what is delivered to students and what is specified by outside bodies, and there is general agreement on what is needed to be an LIS professional on the continent. Outside Europe, there is evidence from other professional bodies that the skills needed as medical search intermediaries apply more generally to other domains as well (Nicholson 2005)—the skills referred to in this paper would also be covered by the three themes found in the European survey.

4 Teaching and learning methods

What are different techniques and methods used in the teaching of IR? The methods usually share common educational goals with methodological approaches used in other disciplines, but also have features that make IR when broken down in to its component parts a very challenging subject to be taught.

We cover the three main areas of teaching: (a) methodological approaches for the classroom, where we wish to show some examples of how to teach IR from a pedagogical perspective; (b) e-learning, in order to present some distance learning experiences with IR; and (c) IRS systems and tools used in teaching and learning IR. We try to give a general vision of the two perspectives involved in this field: teaching and learning, i.e. the delivery and the students’ learning outcomes. A significant finding is that most of the authors who published their research in the IR education field, present their delivery methods in detail (teaching), but do not tackle the learning outcomes for the students (learning). This key point should be taken into account in further research as teaching and learning are complementary aspects.

4.1 Methodological approaches in the classroom

When preparing our teaching material for a course we make a number of assumptions about the level at which we pitch the material to be taught. These assumptions are based on our teaching experience, the level at which the course is aimed at, our goals and expectations, and what we think the students know about the subject. Before commencing delivery, one of the first steps that an instructor should take is to perhaps acquire knowledge about the students in the class and their context. This helps the instructors assess the students’ knowledge of searching, what are their mental models of the entire search process, and what difficulties that they might face. With this information, new methodological approaches can be defined in order to improve student knowledge of the search process.

Some studies have been conducted using this framework. One example is Kuhlthau (1988) who in a major study identified the stages of information problem resolution and then subsequently proposed a model of users’ steps within the problem resolution process: initiation, topic selection, pre-focus exploration, focus formulation, information collection, and search closure. This idea is also utilised by Leide et al. (2007) to present a study of task based searching, the end goal being writing essays. In contrast, work by McGregor (1994) showed that students are “focused on a final product rather than a process”.

These ideas were taken further by Kuhlthau (1997) in order to assist students with the search process. Five strategies were developed for coaching students in search: collaborating (to work with other students), continuing (refinement of the information need), conversing (to talk about what they already know about the problem and what they might be interested in finding out), charting (organizing ideas graphically), and composing (writing all the information gathered).

Related to this idea but from a point of view of the instructor, Cohen (2001) presents ten tips to help teaching web searching. The proposal relies on the fact that it is better to help understand the information need, analyse queries and provide direction to corresponding information sources (different search engines, directories, and deep Web) than to exhaustively show the operation of different search engines. In a later paper, Cohen (2007) defends the usefulness of this query-based approach, but shows, using a study of different teaching resources how it is rarely applied. The outcome of this study from a point of view of student learning outcomes it is that they must learn to understand the information need in detail if they are to take advantage of the search tools. Lazonder (2003) also presents ten simple tips, but to guide lecturers in the process of designing Web search instruction—the main learning output for students is to improve Web searching skills. Vine (2001) describes methods to develop lesson plans for a course on searching in several steps, giving advice about best practice. In this research, planning is the main skills to acquire for the students using the following steps: a) study and analyse the information need; b) choose the source of information (for example, library websites, CD-ROM databases, Web directories, the whole Web; c) select the most appropriate search engine; and d) use it competently.

The two philosophies described above are mapped to the concepts of education and training in terms of searching in the literature. Dimattia (2007) charts the difference in philosophy between these two concepts: “Education focuses on the underlying fundamentals with specific systems as examples. Training focuses on the specific systems and their features”. An alternative view is “theory-based” and “practice-based”: the idea of teaching the underlying theoretical concepts vs. teaching how to use a specific system without understanding the inner workings of it. The argument being that if students understand the underlying principles they can more easily transfer their skills to another system. A clear example of the first approach is found in the context of web-accessible databases: EBSCOhost Academic Elite, FirstSearch EBSCO Master File, Lexis-Nexis Academic Universe, JSTOR and Project Muse (Bernard and Hollingsworth 1999). The starting point for this research was that different web-accessible databases were available with very diverse interfaces and underlying search engines. Should an instructor teach all features of every system, or extract “universal concepts” and teach only those which allow the student to work competently with all systems? The second of these options was preferred. In way the student acquires abstract knowledge of search engine features, knowing the general features of these systems, and thus being able to apply these features to a particular search engine in a given retrieval session.

Strategies are the key concept: students must have a wide range of strategies to tackle the problem of searching, a skill in which they often have little experience in. This is the motivation for the work of Bhavnani et al. (2001) who develop “a framework of effective and general strategies to use complex systems”. In order to reach this final objective they identify a taxonomy of IR tasks, as well as taxonomy of general IR strategies, to design a descriptive model of expert performance and a prescriptive model of effective performance. Using this information, they were able to develop a methodology for helping students learn to search: 1) learn the existence of specific strategies to execute frequent tasks; 2) learn when to use a particular strategy; 3) know how to execute a strategy; 4) learn to use the strategies across applications. The learning outcome here is being able to understand these strategies and when to apply the most appropriate method on a search engine.

Halttunen (2003a) asserts that: “The main goal of instruction is to develop learners’ practical capability to perform successfully any search task occurring in the professional work situation”. A recurrent learning outcome found in the literature is that students’ must have acquired good search skills once IR instruction has finished. Lazarinis (2007) in a more specific context also concludes that teachers have to provide a set of strategies (abilities and knowledge) for the students, allowing them to use of search systems effectively. The author presents an approach based on the Instructional System Design methodology, to design a course for students whose mother tongue is Greek. But as this difficulty is specific to non-native English speakers, the students should be able to recognise the main problems of search engines when different languages are used and how to overcome these problems to search effectively in their given language.

An example of a method to teach search strategies is presented in Walker and Engel (2006). Students are asked to carry out a first research exercise (given a question, they have to provide a short paragraph describing the answers) with some feedback from the instructor. The answers are collected and processed, allowing the instructors to produce a presentation, that analyses the search strategies used and show new ones based on examples. This two step process is repeated twice. The results showed an increasing use of sophisticated search strategies, which lead to better quality of answers. However, some students are not always willing to be taught search (e.g. teenagers who have a very high level of technological literacy), as in Block (2001), so instructors must rethink the explicit teaching and learning strategies and use implicit methods. An example is the development of web sites which are entry points for the concepts and resources to be delivered, so the student is unaware what they are doing while they are acquiring knowledge.

Jones (2007) and this volume noted how the students’ interest increases when the lecturer poses questions instead of providing facts. Assessment therefore plays a very important role in the development of a course, but framed in well designed elements given this viewpoint. The author asserts that it is important to show the objectives of each these elements, and how the assessments could help to achieve understanding of them. In his teaching approach, Jones tries to encourage the students to think critically about the topics contained in each element, rewarding the most creative solutions.

In Halttunen (2003a) and Halttunen (2007), the implementation of two pedagogical approaches, scaffolding and anchored instruction, to the field of IR is presented. The former tries to give some support to learners through their interaction with teachers and tools for search, so they can develop new skills. The latter tries to create “macrocontexts”, i.e. semantically rich environments that integrate concepts in the curricula and set real problems to solve. The author proposes an implementation of the scaffolding technique based on the QPA system (Halttunen 2002; Friman et al. 2005), which gives feedback about the performance of queries allowing the student to learn about formulating queries and improve their knowledge of this task. The application used in the anchored instruction method was search tasks in the context of journalism using newspaper text and image databases. The conclusion of the evaluation was that both techniques could be very useful for IR instruction. Olivier and Olivier (1997) showed that providing context is very important in order to improve the quality of the search, and consequently the results.

As important as it is for a non-specialized student to know how to perform a search strategy, a CS or LIS student needs to know how search engines work. In order to design appropriate teaching and learning methods, an instructor must think about “conceptual approaches, metaphors, representations, and misconceptions” when applying IR tools. Efthimiadis and Hendry (2005) and Hendry and Efthimiadis (2008) elicited this information by getting the students to sketch a diagram showing how they thought a search engine works. The authors concluded that this is a good strategy that improves the students’ conceptual model and technical knowledge of search systems. The tools presented in this research are very useful when designing methods to teach technically focused IR. Analogously, Halttunen (2003b) investigated students’ conceptions of IR know-how exhaustively, and provided some hints to design (constructive) learning environments for IR. This is based on learning outcomes set to focus the student on how to formulate a good query and know and use a wide variety of search techniques, whilst being centred on problem solving and analysis of information sources. Brandt and Uden (Brandt 1997; Brandt and Uden 2003) concluded in their studies that it is important for the students’ to have very “strong mental models” for IR. Otherwise the students (particularly novices) are not likely to be successful at information gathering.

Tomaiulo (1998) argues that, while understanding the students’ mental models is important in order to adapt the instruction according to the findings of a study, there is very little time to perform any study during a course. In this paper, some clues on how to design search sessions are given based on procedural instruction. This educational method is focused on the students work on a computer, rather than lectures and presentations.

The research done by MacFarlane (2007) and this volume focus on the underlying mathematics for IR rather than search strategies. Maths is an important skill for IR students (both CS and LIS), the approach for delivering the subject may enhance learning, particularly when a transmitting model of teaching is used (e.g. the standard lecture format). The author presents some ideas from his personal experience that could improve the knowledge of maths for IR using a transaction model of teaching (creating a two way feedback loop between the instructor and student).

In summary, we suggest that most of the methodological approaches found in the literature are based on the idea that it is better to offer ideas and methodologies for searching rather than starting with the search tools themselves. Instructors usually deliver generic ideas for the IR field, e.g. search strategies and search engine general features, and ask the students to apply them to specific contexts, problems or tools. But the application of these ideas is not done directly. Students should be able to analyse the situations, extract information and useful resources, and apply them appropriately. If this method is successful, then the objectives of the teaching and learning process will be achieved, both in terms of delivery and learning outcomes.

4.2 E-learning and IR instruction

Distance learning using online methods is also a common technique used in the instruction of IR, in its different facets. However we suggest that the development courses using e-learning techniques shares common elements and basic e-learning methodologies from other educational fields (not just computing and information science). We found a large number of papers focused on e-learning in IR in the distance learning literature, but only present some representative examples of different approaches in order to give an idea of general applications of the technology.

Henrich and Morgenroth (2007) present a wide range of distance learning courses on IR, at different levels of education. These successful courses are based on the Moodle platform as well as on DocBook standard for documentation. These have all the typical elements of such a learning style: e-learning material in different formats (mainly, HTML, PDF and MS PowerPoint presentations), communication tools (e-mail, chat, and forums), evaluation modules, etc. They also integrated interactive elements into the technology (e.g. Java applets) with which the student can evaluate and learn methods and algorithms.

In Sacchanand and Jaroenpuntaruk (2006) a “web-based self-training package for information retrieval using the distance education approach” is introduced. The package has been designed and developed using a standard methodology in the field of distance learning. It is composed of three main parts: information about the project, study modules (10 instructional multimedia modules on IR, self-assessments through pre- and post-test) and references and further readings. Delivery modes such as online and offline delivery are provided via the Web and CD-ROM. The package gives a high priority to the development of self-directed information literacy skills among the students.

The GetSmart project (Marshall et al. 2003, 2006) is “aimed at enhancing digital library support for learning processes”. The main feature of this project is to integrate three different but complementary elements into one tool:

  • knowledge construction,

  • digital library, and

  • course management (class administration).

This is to support the information search process. The elaboration of concept maps by students is a key concept in GetSmart and the research demonstrates that the students understanding of material is improved. The project supports three learning activities: individual study of course materials, acquisition of related and additional information and the preparation of group presentations.

A very exhaustive analysis of Computer Assisted Instruction (CAI), applied to search strategies is presented in Fourie (2001), explaining the needs for a CAI-based tutorial on IR. The steps in the design and implementation are discussed, pedagogical considerations, and students’ experiences examined. This is a significant contribution to a formal development of learning systems, which considers a wide range of factors.

4.3 The use of information retrieval systems for teaching purposes

The information retrieval systems (IRS) based instruction is carried out using two different perspectives, largely depending on the type of student. Usually for more technical students, such as CS and specialized LIS students, the IRS’s are an essential component in showing how search engines work internally, with all the details necessary for each discipline. However, for students in other disciplines search is a tool which is used to support learning. We discuss here some of the approaches found in the literature, in these two different areas—specialised and non-specialised.

Starting from the second (non-specialised) perspective, where the IRSs are tools used for learning, commercial systems such as Dialog and Factiva are in widespread use. A number of authors present the advantages of using the Dialog system for teaching search fundamentals (Tenopir 2005; Raban 2003; Drabenstott 2004; DiMattia 2007). Raban (2003) relates six reasons why she finds Dialog useful for teaching search. The central point of the argument is the use of a command language for interacting with the text databases. Tenopir (2005) argues that this method improves knowledge of Boolean search as it isolates the students from the graphical interfaces thereby giving the student an understanding of how an underlying search engine works. The students’ gain a better overall view of search techniques as the commands that they use with this online searching application are available in other types of search systems albeit in different more simplified forms.

Halttunen and Sormunen (2000), Sormunen and Pennane (2004) and Friman et al. (2005) present an alternative using a computer-supported learning environment to teach searching strategies called IR Game (it is also known as Query Performance Analyser (QPA)). This is a system in which common test collections can be used to teach the evaluation and performance of queries. The research suggests that students who complete instruction with the tool improved their “practical capability to perform successfully any search task appearing in the professional work situation”. With the learning tool, they perform searches against a test collection and are given specific feedback on the quality of their searches. More specifically, well-defined search tasks are designed for each database. Students’ then provide relevance judgements to the system. The user with given a search task formulates a query, to which the system responds by providing evaluation measures and visual information about the performance of the query. This approach helps students to develop specialized skills in searching. The authors assert that the students found the feedback from the performance of the query, the ability to re-formulate it, and evaluate the impact on the performance very motivating and improved their learning experience. The tool allows the student to extensively work on the query formulation and improve their abilities for this task. Visualization is also the main topic of the work by Brusilovsky (2002). The author claims that this technique is very useful in IR, and exemplifies its use by means of a system to teach the Boolean model of IR by using graphical examples.

Brajnik et al. (2003) presents FIRE, a collaborative coaching approach for strategic help, i.e. the system provides suggestions to the user on how to formulate a query. The experiments presented showed users lack “an overall strategic view of a search”. It is useful therefore to present students with strategic elements for their search as well as tools that they could use to improve search results.

Focussing in the first (specialised) perspective, the use of complete IRS’s in order to teach the IR process, some tools used in the classroom. One example is the IR Toolbox (Efthimiadis and Freier 2007), “an experiential teaching tool for learning about information retrieval systems”. The student can learn the whole IR process (document analysis, indexing, searching and evaluation), without having to program, at different levels of complexity, and the tool contains individual and group exercises. A second example is IR Base (Calado et al. 2007). This is an object oriented-designed toolkit the aim of which is “integration of components, documentation and services, focused on the rapid development of prototypes for research and teaching”. In this research, students are presented with a wide range of existing classes that show how models could be implemented. The knowledge gained is useful for implementing IR models and for performing experiments with standard test collections. An earlier effort and similar tool was an object oriented IR platform, produced with the aim of providing core functionality to develop new models and algorithms (Wade and Braeckevelt 1994). The learning outcomes in these two last examples are to be able to build a search engine using a set of classes using basic functions, and to develop new modules to include new requirements (for example, new retrieval methods, indexing techniques, etc.).

Another alternative and very useful approach is to use medium or large scale projects (in the sense of the building of a software application) in the course. This is usually done at the end, when all the required concepts have been delivered or alternatively during the course when the lecturer is introducing the key elements of the syllabus. One of the main advantages from this pedagogical point of view of project-driven subjects is that the students get a holistic view of the IR process. When following an assessment approach, the students deal with a specific problem on each occasion in the context of the subject, although each element is usually tackled separately. In the project approach, students’ must deal with all the problems together and approach the different parts of the project as a whole. This means that they must integrate different methodologies and technologies, giving them a different point of view. Real-world scenarios can be presented in these projects. Group projects promote cooperative learning; improve problem-solving abilities and increase knowledge and interpersonal skills, as well as written and oral skills (Chau et al. 2003). This research demonstrates that the students “acquire experience in a variety of aspects, including new technologies, system integration, database administration, and project management” including IR. These kinds of skills are essential in commercial working environments. The authors present their “Build Your Search Engine in 90 Days” method, in which students are asked to build such a system in a semester, with some basic IR building blocks, using requirements specified by the instructors,.

Similar projects to Chau et al.’s work can be found in Meng (2003) and Hendry (2007). The students are asked to design and build a search engine to browse and search resources. As the projects are framed in a wider context, in the field of Digital Libraries, they have a previous modelling and representation task giving them a complete view of a near real project.

Other more generic toolkits from IR research are also used in teaching e.g. Lemur, Lucene, and Terrier (http://www.lemurproject.org/, http://lucene.apache.org/; http://ir.dcs.gla.ac.uk/terrier/, respectively). These systems provide the functionality needed for both the practical and general purpose part of many IR courses (in assessments, projects, or used as examples in class).

Other approaches teach specific IR models by using the underlying IRS’s that implement them e.g. Garnata (de Campos et al. 2007). This system implements the CID model for structured documents (Context-based Influence Diagram model), used to teach doctoral students probabilistic retrieval. The students learn the models based on Bayesian networks and Influence Diagrams, and study the different modules that Garnata provides, as well as the functions for dealing with flat documents that XML retrieval offers. The students can also implement new models supported by the general modules that support XML indexing and management. A further example is a tool designed to teach students the principles and concepts of Fuzzy Information Retrieval Systems based on weighted queries (Herrera-Viedma et al. 2007). This system is not a toolkit, but a Web-based application where the students can formulate weighted queries and the execution of queries are displayed graphically. In addition, the system contains a module that allows test collections to be defined. The authors demonstrate that understanding of Fuzzy IRSs is improved using the tool. Finally, in Jinguji et al. (2006), a system is presented which focused on teaching Question Answering to students undertaking a masters program.

In summary, the development of IRSs from scratch or based on existing technology is a widely used method to teach IR. It helps students to understand the whole process of IR (e.g. elements and interaction) but also it promotes early research in the field.

5 Assessment and feedback

Once a course on any subject has been designed and it is being implemented, it is necessary to test if learning outcomes have been achieved by the students. However any kind of experience using a new teaching method must best, e.g. must assess its impact on students, in order to assess its impact on the desired learning object—its success. The results of these tests can be used by the educator, in order to make the necessary changes which will improve the students learning experience. The perspective of students’ assessment that we want to give here is not just focused on grading their work. Jones (2007) and this volume discusses assessments from this latter perspective and suggests what an assessment of this nature should have.

We present here some examples of assessments in both the course and teaching techniques level as the form a valuable body of prior experience which are useful for instructors of IR. We have found few examples related to the evaluation of the learning outcomes in the relevant literature (Halttunen and Järvelin 2005), and some work outside of the IR field (Whittington and Nankivell 2006; McFarlane 2001) which are from a Computer Graphics Technology Program and ITC respectively.

A first example at course level is presented in Goharian et al. (2004). The authors describe the experience of including a course on IR in their undergraduate computer science curricula. Several exams are set during the course covering the different topics in the syllabus, and the students are surveyed at the beginning and end of it. These tests were designed to assess how much the students thought they knew of the course topics at the beginning, and how much knowledge they thought they have acquired at the end. This research is a good example of using survey techniques to assess the usefulness of assessments.

Zhu and Tang (2006) propose a module-based IR curricular model, based on the design of an array of IR modules, that aims facilitate their adoption and integration. Having designed some courses on IR based on these modules, they assessed their quality by means of objective and subjective assessments. The former is applied before and after taking the corresponding courses, and asks students to indicate their knowledge of IR topics and usage of IR tools. The latter is a set of assignments, projects and tests, in order to measure the students’ objective performance. Using the result of these assessments, instructors can gauge what the students says they have learned, and the compare it to what they actually have learned.

Sacchanand and Jaroenpuntaruk (2006) describe a web-based self-training package for IR. There exist two types of assessments in this research: the first is evaluation of the learning progress by means of self-assessment from pre- and post-tests and exercises at the end of each module; the second is the evaluation of the self-training system itself. Before starting the course a first assessment was carried out by targeted users in a pilot study, to refine and improve the package. Henrich and Morgenroth (2007) report the assessments used were related to evaluate the features of the platform, in the context of an e-learning course.

Hendry (2007) describes ‘History Places’, a team project-based teaching strategy with the aim of implementing a working system in the context of digital libraries. The different editions of the course are assessed by means of a ‘short reflexive statement’ on what the students learned. The students were also asked about what they thought could be modified, added or removed from ‘History Places’. Both sources of information gave valuable information about the experience, what evidence could be used to improve the project. In addition assessments where used to select the team components, depending on student programming skills and other skills directly transferable to the project. Chau et al. (2003) and Meng (2003) also performed an informal evaluation to evaluate the project-based courses. The former in a course used to teach key topics in Computer Science and Information Systems, using the class projects based on Web search; the latter as an appropriate way of acquiring the theoretical aspects of IR.

There are some important examples of specific teaching experiences reported in the literature. One approach used to help students to formulate better queries is IR Game (later QPA) reported in Halttunen and Sormunen (2000), Sormunen et al. (2002) and Friman et al. (2005). The aim of this ‘computer-supported learning environment’ is to ‘demonstrate the performance of queries in different types of search situations’ (Halttunen and Sormunen 2000). It is based on traditional test collections (documents, queries and relevance judgments). Tasks are created around the existing queries, and students are asked to formulate queries on a selected task. The user can observe the performance of her/his queries with respect to the relevance judgments associated to the tasks using a graphical tool. With an exhaustive user evaluation, the authors investigated the learning process and systems functions. The methodology they used to gather data were observation of the instructional design of the lectures supporting system use, observation by recordings the tutored exercises sessions, and finally, stories describing the students’ learning experiences. The analysis of the results gave clues about how the learning process was supported by this tool, identifying features enhancing or inhibiting learning, among other important findings.

Examples of studies centred on the evaluation of learning outcomes using a tool as the basis for instruction can be found in Halttunen and Järvelin (2005) and Halttunen (2003a, 2007). This research is focused on the comparison of traditional and experimental IR learning environments in the context of an introductory subject to IR. The set of assessment tools used included short essays, questionnaires and search logs. Short essays were used at the beginning of the course in conjunction with questionnaires at the end to collect evidence of students IR conception and provide data on learning styles. Search logs of exercises done in tutorials were also analysed to provide more evidence.

A unique way of approaching assessment is presented by Efthimiadis and Hendry (2005) and Hendry and Efthimiadis (2008). The assessment tool was getting the students to draw sketches of how they thought search engines work, the underlying idea being to assess the student knowledge on IR concepts. A reference model of basic IRS components is used in the assessment.

In summary, the assessment of a course or methodology through students’ learning outcomes is of vital importance to improve their learning experience. It is fruitful to assess students’ knowledge of the subject both before and after the learning process, in order to determine its effectiveness. In addition, more objective assessments can be performed, mostly with the aim of comparing the learner’s impression of the process with his or her state of knowledge. In the context of a course, intermediate assessments can be carried out in critical moments of the learning process. This may lead to revision of the course schedule, emphasizing problematic concepts before delivering more material. The nature of the assessment is very varied and depends on the type of information that needs to be obtained.

6 Curricula

There is a significant amount of discussion in the literature about information retrieval curricula, for the main disciplines of concern, library & information science and computer science. The themes in curricula have two focus points: what is to be studied (delivery) and what knowledge/skills the student will have when they have completed the required study (outcomes). Using these focus points we examine the general influence from pedagogy on IR curricula, the issue of interdisciplinarity, the role of professional and accrediting bodies in the development of IR curricula and the development of the subject in both main disciplines.

Firstly what are curricula? Bell and Schauder (2002) refer to the etymology of the term, which “in ancient Greek usage referred to an obstacle course through and over which athletes passed”. The purpose of curricula today is therefore to provide a course composed of “learning experiences and materials (obstacles) provided by educators”. These must clearly be driven by the required educational goals of the particular discipline (see Sect. 3 above). A key question is how is the curricula developed by providing learning objects, which provide the required obstacles? There are many methods proposed for this, and some are mentioned here. Action research has been used for development of curricula (Nicholson 2005; Riding et al. 1995) in order to develop deep learning in the subject together with transferable skills. Sharda (2007) gives an overview of problem and story centred curricula. The development of these techniques is welcome, particularly as many traditional practices are more suitable to the industrial age rather the information age (McFarlane 2003). Also worth mentioning is using either digital library (Gupta et al. 2002) or information retrieval tools to assist the development of curricula.

One particular issue is the extent to which the LIS and CS disciplines share ideas for curricula, to or to put it another way, how interdisciplinary is information retrieval? There are conflicting views on this issue. Rennie (1986) suggest that CS is “computer based researches” whereas LIS is the “mechanisation of library routines using computers” which influences the particular world view, e.g. CS curricula would refer to automatic indexing, whereas LIS would refer to manual indexing via thesauri. Poulter and Brunt (2007) would agree with this view in that the curricula focus in LIS is core skills for librarians, whereas CS looks for understanding of methods for ranking such as tf/idf. There are tensions between these approaches which come about because of the effect of new technologies which cause disintermediation, but which require more IS&R in curricula as a result. Salton (1969) argued that information science concepts in CS are useful—the focus was very much on processing at this stage (Atchison et al. 1968). More recently Croft (2003) in a keynote speech noted that IR has a very strong relationship with LIS, but the CS field is more dynamic and fast moving, which suggests that CS curricula would have to change more often. Saracevic and Dalbello (2001) use an interesting analogy, e.g. Venus vs. Mars—in the same planetary system but moving in different orbits. Their basic argument is that “educational needs differ significantly from education for LIS proper and CS proper” which infers less interdisciplinarity and more specialisation. An alternative view is put forward by Spink and Cool (1999) who argue that the “demand of digital librarians” …”may warrant restructuring of LIS and CS curricula” in order to provide development opportunities in both ‘technical and user aspects’—perhaps ‘moons’ moving around the same planet. Coleman (2002) makes a very strong argument for interdisciplinarity in digital libraries. Further work includes Yang et al. (2006) which takes the CS view (IEEE and ACM 2001), but does include LIS elements e.g. search and evaluation, relevance in context of digital libraries as an important part of DL education, and Riesthuis (2002) who describes an interdisciplinary approach in an LIS department, but with a focus on CS and LIS issues.

Professional and accrediting bodies for CS and LIS have significant input in what goes into curricula, particularly for tertiary education (a very good example is the ACM curricula discussed in the context of educational goals above). The degree to which this is done with respect to IR varies between organisations. The Charted Institute of Library and Information Professionals (CILIP) in the UK, specifies a body of professional knowledge that all Library and Information Scientists must have to be a member of that organisation. The emphasis is very much on what the student can do after study (learning outcomes). This includes knowledge organisation, information needs, service provision and IR as part of an overall target of student information literacy. More specialist knowledge is tackled by Nicholson (2005), who gives a survey of syllabus for medical library searchers (Medical Library Association, USA). In contrast the ACM (USA) has more of an emphasis on delivery (what is to be taught). The ACM attempted a complete map of CS curricula (Atchison et al. 1968), which would make very little sense now given the way the discipline has evolved. Interestingly it included LIS elements of IR such as indexing and classification, but is very system orientated—there is no reference to users. Fox (1996) in a series of workshops attempted to define a set of subjects for CS and IR in the context of ACM SIGIR. More recently in IEEE and ACM (2001), the emphasis has changed to practical search skills, with more of a focus in HCI for IR in such fields as Multimedia. Both undergraduate curricula for Information Systems (ACM and AITP 1997; ACM et al. 2002) have very little reference to IR. ACM and AITP (1997) refers only to ‘External DB retrieval’ which could be a reference to online databases, whilst ACM et al. (2002) refers only to search strategies in the context of retrieval for personal productivity. The British Computer Society (BCS) of the UK does not explicitly reference IR in its documentation. However the BCS does require group work, which fits in with Chau et al.’s (2003) work. Other types of organisations are also involved. The UK Quality Assurance Agency for Higher Education (QAA) has a number of subject benchmarks including ones for both CS and LIS. The CS benchmark specifies a set of generic computing skills to design and build IR systems together with transferable skills (doing search) and a set of specific topics for IR. The LIS benchmark has more of a focus on IR, with subjects such as searching and analysing information needs having a high profile (these are listed in more detail in Sect. 3.2). CILIP and the QAA have collaborated in order to produce the LIS benchmark, whereas there is no direct evidence of the BCS’s involvement in the CS benchmark. This provides some evidence (for the UK at least) of far more interest by the LIS community in IR than the CS community.

Instructors also develop their own curricula from a variety of other sources other than professional bodies. Research Universities in the US require that instructors integrate research in to their teaching. This is done by requiring research based readings, asking students to do research-based term projects, etc. Turning first to the CS field there are a number of examples of this source from the literature. Croft (2003) mentions that it was not uncommon for many early CS departments to have facility working on IR and that this had an impact on subject recommendations for IR on CS courses. Goharian et al. (2004) focuses on teaching developing efficient programs for retrieval and relevance feedback, using underlying index structures and models for search. Zhu and Tang (2006) lists a wide number of subjects obtained from research in the field, taking general IR concepts and adapting them to different requirements. Goharian et al. (2004) note the distinction between Bachelors and Masters education, the latter requirement more mathematical maturity in the context of IR. Wing (2000) outlines ways of weaving formal methods into the CS undergraduate curricula e.g. induction, specification and verification etc, which are useful for understanding IR models and theory. Sharda (2007) puts forward a general point about what should be taught in CS undergraduate courses—with respect to the rather traditional and outdated idea of the ‘program as an island’ which they assert is not relevant today. This point is significant in our context, as it is clear that IR modules need to impart the understanding of interaction with users—clearly ‘no program is an island’, the field either innovates or dies. Chau et al. (2003) suggests that the industry requirements for CS skills (design and technical) can be embodied in a search engine group project, and that such projects are a good way to integrate these required computing skills (which may fulfil requirements of accrediting bodies as well—see above). Pomerantz et al. (2007) note that there are many fewer CS programs with a Digital Library focus than LIS (Pomerantz et al. 2006a). In general the focus in the CS literature appears to be mostly on delivery.

It comes as no surprise therefore that when we consider curricula in LIS sources; there is a much greater emphasis on IR in this field. The evidence for this is rather strong (see Sect. 3.2 on LIS educational goals). Bates et al. (2005) surveyed information seeking and retrieval (IS&R) using a forum as part of a bigger survey of LIS curricula in Europe (Kajberg and Lørring 2005). The study has many themes, but only (IS&R) has 100% representation on all courses in the survey. It will have been noted from the educational goals identified in this study (see Sect. 3.2) that the focus of it was quite broad, and included human information behaviour. However, it is clear that in Europe IR is considered to be the essential component of LIS study. The study provides some important evidence on the feedback loop between professionalism and curricula, and the important effect research in the field has on this loop. The design of degree curricula as a whole as well as module/course is discussed in this research. Further information can be found in Bawden et al. (2005) and Bawden et al. (2007). Vilar et al. (2007) build on this work, but apply it to programmes in two information schools, Ljubljana and Dublin, and look at different levels e.g. Bachelors and Masters. Nicholson (2005) deals with this issue in North America, discussing whether IR should be a core or elective module, coming down firmly on the side of core, suggesting that “the amount of time dedicated to search education in the core courses should be reconsidered to ensure that library schools are preparing their students to survive as information professionals in the increasingly digital information future.”

Digital Libraries (DL) are a broader topic than IR, but IR is an important component of DL education. Choi and Rasmussen (2006) looked at current practice in academic libraries in order to inform the development of curricula for Digital Library courses. Only a small part of the responsibilities to feed into skills required are mentioned e.g. 8.57% on processing (searching and metadata). Pomerantz et al. (2006a) provide a theoretical framework, which is used in Pomerantz et al. (2006b) using citations provided on published syllabi accredited by the ALA (USA) to build topics, and validate this against IEEE and ACM (2001). The focus here is on DL research rather than ALA requirements. Topics include information and knowledge organisation and services, search and browsing. Ma et al. (2006) also surveys DL curricula, noting that there is a “lack of information professionals with the right combination of skills” which is particularly acute in the area. Research in this area is being jointly pursued by the University of North Carolina at Chapel Hill and Virginia Tech as part of the Digital Library Curricula Development project (DLCD 2008)—this project is ongoing at the time of writing, and readers can refer to new materials as they are produced.

Looking at other sources for curricula design, Brower (2004) discusses the involvement of students in developing curricula, and LIS skill for other subjects. Fjallbrant (1996) also addresses the latter for building a larger course in search to support information literacy outside the field, for end users who are arts or science students. Information literacy is also the focus of Kaplan et al. (2004), which looks at search and browsing to assist children build skills for the broader LIS area. Druin et al. (2007) looking at DL support for pre-university education found that children preferred physical interactions, but would still use online search tools. Pollit (1987) looked at pre-university education in the U.K., reviewing the skills needed at that stage. Interestingly he also defined a 4th R along with ‘Reading’, ‘wRiting’ and ‘aRithmetic’, namely ‘Retrieval’.

This section gave an overview of who decides what goes into curricula, why it goes in, when it goes in and how it goes in. From the literature it is apparent that professional and accrediting bodies feed their requirements into IR subjects, with varying degrees of specificity for both main fields. These requirements tend to be outcome focused. It is clear that instructors own research either though the literature or through work carried out directly also feed into courses. In contrast this source of information for IR courses tends to be delivery focused. What goes into curricula does depend on the discipline, with CS focused more on the design and building of IR systems, with LIS courses more focused on information needs, seeking and search. It is clear from the literature that LIS have a much broader coverage of IR than CS, but that CS subject moves quicker due to changes in technology. When IR should be taught depends on the need of the student. Pre-University students benefit from some training in information literacy, undergraduate CS students benefit from the varied ideas from computing which make up IR and LIS postgraduate study requires knowledge of search—these are but a few examples. There is one final issue—should education be specialist or generalist e.g. (Nicholson 2005) has a focus on the medical domain. The conclusion has to be that generalist is preferable, as the skills obtained through such a route is more likely to be transferable.

7 Citation survey

In preparation for the review paper the authors conducted extensive literature searches. The aim was to cover the literature on teaching and learning in IR as broadly as possible. The search included databases, such as INSPEC, ERIC, digital libraries, such as, ACM-DL, PROQUEST, EBSCO, electronic journals, such as DLIB, Information Research, and references sent to the authors by researchers in response to a call for citations in cognate listservs.

A total of 159 articles were identified that had anything to do with teaching and learning of information retrieval or search in general covering the period 1968–2008. These papers were analyzed at both the “work” level and the “citation” level. At the “work” level each paper was examined to establish the technical level of treatment of the subject covered (non-, medium-, highly-technical). The educational goals and cognate domains were identified. The teaching methods were categorized. Finally, the “work” was examined to establish the “education” area it targeted, for example, general education field, LIS or CS education subfield, related to education but not directly targeting an education field.

Of the 159 works, 85.5% were from the LIS/CS fields, 5.7% from the general education field and the remaining 8.8% were unrelated to education. The technical level discussed at these works ranged from 46.5% non-technical, to 37.1 mid-technical, and 16.4 high-technical levels. The works were mostly split between the LIS field (48.4%) and CS (43.4%). The remaining was split between MIS, linguistics, psychology, and other.

The methods discussed in the 159 works ranged from curriculum (27%), classroom (26.4%), e-learning (13.8%), IR theory (11.9%), modelling (11.9%), and assessment (8.2%).

The analysis at the “citation” level included the following. For each of the 159 papers the cited references were examined individually and were categorized as to whether they were citations from the field of education, from the LIS or CS education subfield, or they were general citations unrelated to education. The total number of citations listed in the 159 works is 3054. This breaks down to an average of 19.2 references per work, and a median of 13, with a min of zero and a max of 103. Of the total number of citations about 11.1% of the citations were from the field of education, 14.1% from the fields of LIS/CS, and the remaining from other areas.

This citation study provides a snapshot of the trends in IR education and tells us of how and whether IR educators are informed by education principles in their teaching of the subject. The results demonstrate that currently there is a small (about 11%) but increasing interest in educational theory and its application to IR. This does not mean that IR educators do not care about their teaching. It means that there is a wide open opportunity for the IR educators to tap into the established work of educational theory and apply these approaches to IR teaching and learning.

8 Summary and conclusion

Using a taxonomy which covers both IR based teaching literature and the pedagogical literature, we have shown that research on teaching and learning has made an impact on the delivery of the subject to students. In conjunction we present a number of ideas, which have been tested in real situations, which readers can consider using to inform their own teaching. Examples of these include pre- and post-experience questionnaires to collect student knowledge in the area, and the effect a module or course has had on their learning. A clear issue arising from the literature on educational goals and curricula design is the focus of the teaching—is it to successfully deliver the material to the students, or is it to ensure that students are able to complete a set task after a module. The former is delivery focused (teaching) and the latter is outcome focused (learning). From a student centred viewpoint, it is clear that the outcome based strategy is preferable. Delivery while being important is only part of the solution to the problem of teaching students the subject. IR educators are involved in very innovative teaching methods, using various software tools in teaching and learning methods to inform assessment and feedback. The citation study of the literature shows an increasing interest in taking pedagogical research and applying to the specific case of information retrieval. This trend and the considerable interest in teaching shown by IR instructors are of great benefit to students and their experience of the subject.

A reader, who is interested in best practice in the field, will find this review a useful resource to inform their own teaching. While not by any means exhaustive, the material presented represents an impressive body of work. What informs education will be of particular interest. Readers may look to their own professional bodies for guidance, and may indeed use their own research as the basis for teaching. We have found evidence in the literature for both of these strategies. It is clear however that further work in the teaching and learning in IR is necessary to give a much clearer picture of the field. Given this a more sophisticated citation study is merited to examine the impact of teaching in IR. There is a clear opportunity to investigate the general pedagogical literature to improve teaching and learning in this increasingly important subject.