Abstract
The main objective of this work is to analyse the contributions of Judit Bar-Ilan to the search engines studies. To do this, two complementary approaches have been carried out. First, a systematic literature review of 47 publications authored and co-authored by Judit and devoted to this topic. Second, an interdisciplinarity analysis based on the cited references (publications cited by Judit) and citing documents (publications that cite Judit’s work) through Scopus. The systematic literature review unravels an immense amount of search engines studied (43) and indicators measured (especially technical precision, overlap and fluctuation over time). In addition to this, an evolution over the years is detected from descriptive statistical studies towards empirical user studies, with a mixture of quantitative and qualitative methods. Otherwise, the interdisciplinary analysis evidences that a significant portion of Judit’s oeuvre was intellectually founded on the computer sciences, achieving a significant, but not exclusively, impact on library and information sciences.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
A customized profile including the 47 contributions was created for the occasion. Duplicate records were appropriately merged to gather all citations covered by Google Scholar database.
Journal of Computer-Mediated Communication: according to Scopus, this journal is categorized under Computer Science. In this work, ‘Social sciences’ category was added; Plos One: according to Scopus, this journal is categorized under Agricultural and Biological Sciences, Medicine, Biochemistry, Genetics and Molecular Biology. In this work, it was categorized under ‘Multidisciplinary’. Science: according to Scopus, this journal is categorized under Multidisciplinary and Arts and Humanities. In this work, only ‘Multidisciplinary’ was considered.
References
Bar-Ilan, J. (1998a). On the overlap, the precision and estimated recall of search engines. A case study of the query “Erdos”. Scientometrics,42(2), 207–228. https://doi.org/10.1007/bf02458356.
Bar-Ilan, J. (1998b). The mathematician, Paul Erdos (1913–1996) in the eyes of the Internet. Scientometrics,43(2), 257–267. https://doi.org/10.1007/bf02458410.
Bar-Ilan, J. (2000). The web as an information source on informetrics? A content analysis. Journal of the American Society for Information Science and Technology,51(5), 432–443. https://doi.org/10.1002/(sici)1097-4571(2000)51:5%3C432:aid-asi4%3E3.0.co;2-7.
Bar-Ilan, J. (2001). Data collection methods on the web for informetric purposes: A review and analysis. Scientometrics,50(1), 7–32.
Bar-Ilan, J. (2002). Methods for measuring search engine performance over time. Journal of the American Society for Information Science and Technology,53(4), 308–319. https://doi.org/10.1002/asi.10047.
Bar-Ilan, J. (2003). Search engine results over time: A case study on search engine stability. Cybermetrics,2/3, 1–16.
Bar-Ilan, J. (2005a). Expectations versus reality—Search engine features needed for Web research at mid 2005. Cybermetrics,9, 1–26.
Bar-Ilan, J. (2005b). Expectations versus reality—Web search engines at the beginning of 2005. In Proceedings of ISSI 2005: 10th international conference of the international society for scientometrics and informetrics (Vol. 1, pp. 87–96).
Bar-Ilan, J. (2010). The WIF of Peter Ingwersen’s website. In B. Larsen, J. W. Schneider, & F. Åström (Eds.), The Janus Faced Scholar a Festschrift in honour of Peter Ingwersen (pp. 119–121). Det Informationsvidenskabelige Akademi. Retrieved 15 January 15, 2020, from https://vbn.aau.dk/ws/portalfiles/portal/90357690/JanusFacedScholer_Festschrift_PeterIngwersen_2010.pdf#page=122.
Bar-Ilan, J. (2018). Eugene Garfield on the web in 2001. Scientometrics,114(2), 389–399. https://doi.org/10.1007/s11192-017-2590-9.
Bar-Ilan, J., Mat-Hassan, M., & Levene, M. (2006). Methods for comparing rankings of search engine results. Computer Networks,50(10), 1448–1463. https://doi.org/10.1016/j.comnet.2005.10.020.
Thelwall, M. (2017). Judit Bar-Ilan: Information scientist, computer scientist, scientometrician. Scientometrics,113(3), 1235–1244. https://doi.org/10.1007/s11192-017-2551-3.
Author information
Authors and Affiliations
Corresponding author
Additional information
This paper is dedicated to the memory of Judit Bar-Ilan (1958–2019), an outstanding scholar and an inimitable friend and colleague.
Appendices
Appendix 1: Bibliographic corpus (n = 47 contributions)
ID | Title | Source | Citations (GS) | Citations (Scopus) | Year |
|---|---|---|---|---|---|
p001 | On the overlap, the precision and estimated recall of search engines. A case study of the query “Erdos” | Scientometrics | 51 | 28 | 1998 |
p002 | Search engine results over time: A case study on search engine stability | Cybermetrics | 199 | 88 | 1998 |
p003 | The life span of a specific topic on the web: the case of “informetrics”: A quantitative analysis | Scientometrics | 50 | 23 | 1999 |
p004 | Evaluating the stability of the search tools Hotbot and Snap: a case study | Online information review | 51 | 23 | 2000 |
p005 | The Web as an information source on informetrics? A content analysis | JASIS | 82 | 39 | 2000 |
p006 | Data collection methods on the Web for infometric purposes—A review and analysis | Scientometrics | 180 | 89 | 2001 |
p007 | How much information do search engines disclose on the links to a web page? A longitudinal case study of the ‘cybermetrics’ home page | Journal of information science | 44 | 19 | 2002 |
p008 | Criteria for Evaluating Information Retrieval Systems in Highly Dynamic Environments | CEUR Workshop Proceedings | 7 | 0 | 2002 |
p009 | Methods for measuring search engine performance over time | JASIST | 117 | 52 | 2002 |
p010 | How do search engines handle non-English queries? A case study. | WWW (Alternate Paper Tracks) | 29 | 2003 | |
p011 | Evolution, continuity, and disappearance of documents on a specific topic on the web: A longitudinal study of “informetrics” | JASIST | 79 | 50 | 2004 |
p012 | Dynamics of Search Engine Rankings-A Case Study. | WebDyn@ WWW | 14 | 2 | 2004 |
p013 | Search engine ability to cope with the changing web | Web dynamics | 32 | 2004 | |
p014 | The use of Web search engines in information science research | Annual Review of Information Science and Technology (ARIST) | 136 | 71 | 2004 |
p015 | Comparing rankings of search results on the web | Information Processing & Management | 109 | 43 | 2005 |
p016 | From the search problem through query formulation to results on the web | Online Information Review | 25 | 8 | 2005 |
p017 | How do search engines respond to some non-English queries? | Journal of Information Science | 75 | 38 | 2005 |
p018 | Expectations Versus Reality—Web Search Engines at the Beginning of 2005 | Proceedings of ISSI 2005 | 2 | 1 | 2005 |
p019 | Expectations versus reality—Search engine features needed for Web research at mind | Cybermetrics | 61 | 31 | 2005 |
p020 | Tauglichkeit von Suchmaschinen für deutschsprachige Abfragen: Schwerpunktthema Suchmaschinen | Information-Wissenschaft und Praxis | 7 | 4 | 2005 |
p021 | Mark Levene An Introduction to Search Engines and Web Navigation. Addison Wesley, Pearson Education (2006). ISBN 0-321-30677-5.£ 39.99. 365 pp. Softbound | The Computer Journal | 0 | 2006 | |
p022 | Methods for evaluating dynamic changes in search engine rankings: a case study | Journal of Documentation | 17 | 9 | 2006 |
p023 | Web links and search engine ranking: The case of Google and the query “jew” | JASIST | 25 | 18 | 2006 |
p024 | False Web memories: A case study on finding information about Andrei Broder | First Monday | 5 | 3 | 2006 |
p025 | Methods for comparing rankings of search engine results | Computer networks | 161 | 82 | 2006 |
p026 | Analysis of queries reaching SHIL on the web—an information system providing citizen information | International Workshop on Next Generation Information Technologies and Systems | 0 | 0 | 2006 |
p027 | Popularity and findability: Log analysis of search terms and queries for public services | ILAIS 2006 Conference | 0 | 2006 | |
p028 | Position paper: access to query logs—an academic researcher’s point of view | Query Log Analysis Workshop, WWW | 25 | 2007 | |
p031 | Manipulating search engine algorithms: the case of Google | Journal of Information, Communication and Ethics in Society | 26 | 13 | 2007 |
p032 | Popularity and findability through log analysis of search terms and queries: the case of a multilingual public service website | Journal of Information Science | 25 | 14 | 2007 |
p033 | User rankings of search engine results | JASIST | 66 | 42 | 2007 |
p034 | The lifespan of “informetrics” on the web: an eight year study (1998–2006) | Proceedings of ISSI 2007 | 0 | 2007 | |
p036 | The lifespan of “informetrics” on the web: an eight year study (1998–2006) | Scientometrics | 49 | 25 | 2009 |
p037 | A method for measuring the evolution of a topic on the Web: The case of “informetrics” | JASIST | 18 | 13 | 2009 |
p038 | Topic-specific analysis of search queries | Proceedings of the 2009 workshop on Web Search Click Data | 22 | 8 | 2009 |
p039 | Users’ views on country specific search engine results | Proceedings of the ASIST | 0 | 0 | 2009 |
p040 | Presentation bias is significant in determining user preference for search results—A user study | JASIST | 77 | 46 | 2009 |
p041 | A method to assess search engine results | Online Information Review | 16 | 9 | 2011 |
p042 | The impact of task phrasing on the choice of search keywords and on the search process and success | JASIST | 24 | 11 | 2012 |
p043 | Search Engines and Hebrew-Revisited | Language, Culture, Computation. Computing-Theory and Technology | 0 | 0 | 2014 |
p045 | How and why do users change their assessment of search results over time? | Proceedings of the ASIST | 4 | 1 | 2015 |
p046 | Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings | Aslib Journal of Information Management | 6 | 4 | 2016 |
p048 | A Markov chain model for changes in users’ assessment of search results | PloS one | 3 | 3 | 2016 |
p049 | Analysis of change in users’ assessment of search results over time | JASIST | 3 | 3 | 2017 |
p050 | Categorical relevance judgment | JASIST | 1 | 1 | 2018 |
p051 | Eugene Garfield on the web in 2001 | Scientometrics | 0 | 0 | 2018 |
p052 | Data Collection from the Web for Informetric Purposes | Springer Handbook of Science and Technology Indicators | 0 | 0 | 2019 |
Appendix 2: Systematic analysis: indicators measured, methods employed, search engines covered, queries analysed and sample sizes
Article ID | Indicators measured | Method | Search engine | Queries analysed | Sample | Rounds |
|---|---|---|---|---|---|---|
P001 | Precision; Technical precision; Estimated recall; Overlap; Coverage; Evolution | Informetrics | Altavista; Excite; Infoseek; Lycos; Magellan; Opentext | 1 query: Erdos | 6681 URLs | 6 rounds. monthly Nov 1996 to Dec 1997 |
Coverage; Overlap | Informetrics | Altavista; Excite; Hotbot; Infoseek; Lycos; OpenText | 1 query: Bibliometrics AND growth | 146 URLs | ||
P002 | Coverage; Evolution; Relative coverage; Total relative coverage; Technical precision; Technical relevance; Fluctuation (URL Recovery; URL Permanence); Self-Overlap | Informetrics | Altavista; Excite; Hotbot; Infoseek; Lycos; Northern Light | 1 query: Informetrics OR informetric | 1268 URLs | 5 rounds. monthly Jan to Jun 1998 |
P003 | Fluctuation; Change type (minor and considerable); Change stability (stagnant and dynamic) | Content Analysis Informetrics | Altavista; Excite; Hotbot; Infoseek; Lycos; Northern Light | 1 query: Informetrics or informetric | 1268 URLs | 6 rounds. monthly Jan to Jun 1998 |
P004 | Coverage; Query size; Query type; Technical precision; Fluctuation (lost URLs, recovered URLs, Dropped URLs) | Informetrics | Hotbot; Snap’s Power Search | 20 queries: WebFerretPro; last total eclipse of the Millenium; “Erich Segal” + Doctors; “existential therapy” AND NOT (anxiety OR psychotherapy); http://sites.huji.ac.il/IFLA2000/66intro.htm; protochlorophyllide; Colima Volcano; onomatopoeia + Japanese; non-repudiation AND NOT (privacy OR security); http://www.altavizsla.matav.hu; caprylic; Lawrence Olivier; “Six Day War” + Golan; (“chinese noodles” OR “chinese fried rice”) AND NOT pork; http://www.neci.nj.nec.com/homepages/lawrence/; Nabucco; Charlie Daniels Band; Teletubbies + Dipsy + “Tinky Winky”; (“citation analysis” OR “co-citation analysis”) AND NOT ISI; http://www.huji.ac.il | NA | Daily Sep to Oct 1999. |
P005 | Coverage; Precision; Multiplicity; Recall | Content Analysis | Altavista; Excite; Hotbot; Infoseek; Lycos; Northern Light | 1 query: Informetrics OR informetric | 942 URLs | 1 round Jun 1998 |
P006 | Coverage | Informetrics | Altavista; Northern Light; Hotbot; Fast | 8 queries: ccTLD:.br;.nl gTLD:.com,.edu,.org,.gov,.net and.mil). | NA | 1 round 2 Sep 2000 |
Altavista Northern Light | 3 queries: industry AND government.; university AND government.; university AND industry AND government | |||||
Altavista; Northern Light | 2 queries: “University” (Netherlands) “Industry” (Netherlands | |||||
Coverage; Relevance; Self-Overlap | Google; Webtop; Altavista; Fast; Northern Light; Iwon; Snap | 1 query: Webometrics | 308 URLs | |||
P007 | Coverage (link pages; concealed pages); Technical Precision | Content Analysis Informetrics | Altavista; Raging Search; Fast; Google; Hotbot; Iwon; Northern Light | 1 LINK DOMAIN query per search engine: link:www.cindoc.csic.es/cybermetrics/cybermetrics.html Several LINK URL queries like | 456 total URLs | 4 rounds Jan 2001 to Jan 2002 |
P009 | Coverage; Relative coverage; Technical Precision; Fluctuation; Self-overlap | Informetrics | Altavista; Excite; Fast; Hotbot; Google; Northern Light | 1 query: aporocactus | NA | 33 rounds. weekly and monthly Jan 2000 to Jan 2001 |
P010; P017 | Coverage | Informetrics | Yandex; Rambler; Aport | 9 queries in Russian: Oкнo; Oкoн; бeльıй; Бeльıй; чeлoвeк шeл; люди идyт; люди идyт; нaчинaть; нaчaть | NA | 1 round Nov 2002 |
Voila; AOL France; La Toile | 5 queries in French Electricite; électricité; l’électricité; cheval; chevaux | |||||
Origo-Vizsla; Startlap; Heureka | 8 queries in Hungarian Kar; kár; kutya; kutyák; falu; falvak; javítás; kijavítás | |||||
Morfix; Walla | 8 queries in Hebrew [universita]; [hauniversita]; [bauniversita]; [universitat]; [veshehauniversita]; [mehabait]; [bait]; [midbar/medaber/midavar] | |||||
Altavista; Fast; Google | 30 queries (in each of the languages) | |||||
P011 | Coverage; Growth rate (evolution); Fluctuation (URL Modification, URL Disappearance, URL Persistence) | Content Analysis | Altavista; Excite; Hotbot; Infoseek; Lycos; Northern Ligh; Fast; Google; Teoma; Wisenut | 1 query: Informetrics OR informetric | 7063 URLs | 4 rounds. yearly 1998, 1999, 2002, 2003 |
P012 | Coverage evolution; Overlap; Self-overlap; Results rank | Informetrics | Google.com; Google.co.uk; Google.co.il; Alltheweb | 10 queries Modern architecture; Web data mining; World rugby; Web personalization; Human cloning; Internet security; Organic food; Snowboarding; DNA evidence; Internet advertising techniques | 27 users | 2 rounds. twice a day Oct 2003 to Jan 2004 |
P015 | Rank overlap | Informetrics | Google; Alltheweb; Altavista; Hotbot | 15 queries | 16,985 URLs | 1 round Dec 2003 |
P016 | Search instructions; query formulation | User study | No specific search engine | 178 queries | 35 users | 1 round May 2003 |
P018; P019 | Domain Coverage | Informetrics | Google; Yahoo; MSN Beta | 4 queries: ccTLD:.hu;.ca;.dj;.sr | NA | 1 round. Jan 2005 |
P022 | Overlap; Self-overlap; Results rank; Change average ranking | Informetrics | Google; Alltheweb | Same Record P012. | NA | 2 rounds. twice a day Oct 2003; Jan 2004 |
P023 | Link page characteristics; Link characteristic; Rank position; Link features | Content Analysis | 1query: ‘jew’ | Site1: 689 pages Site2: 294 pages | 1 round Aug 2004 | |
P024 | Search tasks | User study | Google; Altavista; Alltheweb; Teoma; Yahoo; MSN | 2 queries: andrei broder andrei broder bio | 49 participants 1 page | 1 round May 2005 |
P025 | Overlap; Self-overlap; Rank variability | Informetrics | Google; Yahoo; Teoma; Google Images; Yahoo images; Picsearch | 5 queries US elections 2004; DNA evidence; Organic food; Twin towers; Bondi beach | NA | 2 rounds. once a day Nov2004; Feb 2005 |
P026; P027; P032 | Query syntax; Query frequency; Query length; Query output; Query evolution; Queries from search engines | Content analysis Web-log analysis | No search engine | 266,295 queries | 1 site: | 1 round Mar 2005 to Oct 2005 |
P033 | Ranking overlap; User ranking; USER –SE Similarity; Popularity; Relative relevance | Informetrics User study | Google; MSN; Yahoo | 12 queries ‘search engine coverage’; Glycemix index; “web preservation”; Genetic engineering; Stop smoking; Blood test Indexing; Semantic web; Bird flu; Ranking metasearch; Atkins diet | 67 participants 120 results | 3 week long round Nov 9 to 29, 2005 |
P036 | Coverage; Coverage evolution; URL persistence | Content Analysis Informetrics | Altavista; Excite; Hotbot; Infoseek; Lycos; Northern Light; Google; Teoma; Wisenut; Gigablast; Yahoo; Exalead; MSN | 4 queries: Informetrics or informetric; informetrics-scientometrics; informetrics scientometrics; informetrics site:.es –filetype:pdf | 36,282 URLs | 7 rounds. yearly 1998; 1999, 2002, 2003, 2004, 2005, 2006 |
P037 | Technical relevance URL intermittence; URL lost; URL forgot; URL recovered | Informetrics | Altavista; Excite; Hotbot; Infoseek; Lycos; Northern Light; Google; Teoma; Wisenut; Gigablast; Yahoo; Exalead; MSN | 1 query: Informetrics or informetric | NA | 7 rounds. yearly 1998; 1999, 2002, 2003, 2004, 2005, 2006 |
P039; P041 | Ranking Overlap; User Ranking overlap; SE-User similarities | User study | Google (Google.com Google.co.uk Google.co.il) Live Search (live.com; UK search; Israel search) | 9 queries: [Social Networks facebook]; [Hilary Clinton]; BMI; Israel; [Skin cancer prevention]; [html for beginners]; [Olimpics Beijing]; [World Health Organization]; [Google new developments] | 283 total URLs 24 users | 2 stages. July 2008 |
P040 | Rank order user preference | User study Questionnaire | Google; Windows Live; Yahoo | 13 queries: Anthrax; Making money on the internet; Plasma versus LCD; Prague tourist sights; Rembrandt; Ronaldinho; Calculating Page Rank; Search optimization; Free antispyware; Sudoku; Andrei Broder; Louvre map | 120 results 65 users | 1 round October 2006 |
P043 | Coverage; Freshness | Informetrics | Google (google.co.il); Walla; Morfix; MSN; Tapuz; Yahoo | 15 queries: [university]; [universities]; [The university]; [to the university]; [in the university]; [from the university]; [The university OR of the university OR in the university]; [University OR universities OR the university OR to the university OR in the university OR from the university OR university of]; [Library] two spelling variants; [recipes]; [recipe]; [the recipes]; [cellphones]; [cellphone] two spelling variants; [Western Galilee College] two spelling variants | NA | 1 round July 2007 |
P042 | Search tasks | Questionnaire Log files User study | 4 tasks: Task Online Spending; Task Financial concern; Task Children; Task bank | 100 users 88 log files | 1 round Jun to Jul 2007 | |
P045 | User ranking relevance | User study | 1 query: “cyber warfare” | 20 results 35 individuals | 3 rounds n.d. | |
P046; P049 | User ranking relevance; User ranking relevance change; URL rank; User-SE rank overlap; Coarseness; Locality | User study | Bing | 2 queries: Big data [Alzheimer] in hebrew | 20 URLs per query 87 users | 2 rounds n.d. |
P048 | Rank relevance change | User study | Bing | 3 queries: Big data [Alzheimer] in hebrew “cyber warfare” | 120 users | 2–3 rounds n.d. |
P049 | Category-based relevance; Average concordance; Swap ratio | User study | 2 queries: Atkins diet Cloud computing | Sets of 20 results 86 users | 3 rounds N.d. | |
P051 | Coverage; Link pages categorization | Content Analysis | Altavista; Fast; Google Hotbot; Northern Light | 5 queries: ‘Eugene Garfield’; ‘Garfield Eugene’; ‘Gene Garfield’; ‘E. Garfield’; ‘Garfield E’ | 4120 URLs gathered 1073 URLs analysed | 1 round August 2011 |
P052 | Coverage | Informetrics | Google; Bing; Yahoo | 26 queries: gTLP:.com;.org;.edu;.net,.gov;.mil ccTLP:.uk,.ca.;.au;.nz.;.es;.fr;.de;.il;.cn;.ru;.br;.za Yahoo Altavista; Yahoo AND Altavista; Altavista Yahoo; Altavista AND Yahoo; Altavista; Yahoo; Altavista OR Yahoo; Altmetrics | NA | 1 round December 2017 |
Rights and permissions
About this article
Cite this article
Orduña-Malea, E. Crossing the academic ocean? Judit Bar-Ilan’s oeuvre on search engines studies. Scientometrics 123, 1317–1340 (2020). https://doi.org/10.1007/s11192-020-03450-4
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03450-4