Payoffs and pitfalls in using knowledge-bases for consumer health search

Abstract

Consumer health search (CHS) is a challenging domain with vocabulary mismatch and considerable domain expertise hampering peoples’ ability to formulate effective queries. We posit that using knowledge bases for query reformulation may help alleviate this problem. How to exploit knowledge bases for effective CHS is nontrivial, involving a swathe of key choices and design decisions (many of which are not explored in the literature). Here we rigorously empirically evaluate the impact these different choices have on retrieval effectiveness. A state-of-the-art knowledge-base retrieval model—the Entity Query Feature Expansion model—was used to evaluate these choices, which include: which knowledge base to use (specialised vs. general purpose), how to construct the knowledge base, how to extract entities from queries and map them to entities in the knowledge base, what part of the knowledge base to use for query expansion, and if to augment the knowledge base search process with relevance feedback. While knowledge base retrieval has been proposed as a solution for CHS, this paper delves into the finer details of doing this effectively, highlighting both payoffs and pitfalls. It aims to provide some lessons to others in advancing the state-of-the-art in CHS.

Appendices

Appendix 1: Statistical significance analysis

See Tables 16, 17, 18, 19, 20, 21, 22, 23, 24, 25 and 26.

Table 16 Statistical significance analysis for results in Table 3: Choice 1. n, b, and r mark statistical significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 17 Statistical significance analysis for results in Table 4: Choice 2. n, b, and r mark statistical significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 18 Statistical significance analysis for results in Table 5 (top): Choice 3 - all queries set. n, b, and r mark statistical significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 19 Statistical significance analysis for results in Table 5 (bottom): Choice 3 - high coverage queries set. n, b, and r mark statistically significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 20 Statistical significance analysis for results in Table 6 Choice 4. n, b, and r mark statistically significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 21 Statistical significance analysis for results in Table 7 (top): Choice 5 - All queries set. n, b, and r mark statistically significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 22 Statistical significance analysis for results in Table 7 (bottom): Choice 5 - high coverage queries set. n, b, and r mark statistically significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 23 Statistical significance analysis for results for CLEF 2015 obtained using the best settings on CLEF2016 in Table 12. n, b, and r mark statistically significant differences (pairwise t-test with Bonferroni correction, ${p} < 0.05$) for nDCG@10, bpref, and RBP@10, respectively

Table 24 Statistical significance between results of the CLEF2016’s best settings using CLEF2016-2017 validation data in Table 13 (top): the all queries set. n, b, and r show statistically significant ($\hbox {pairwise bonferroni} < 0.05$) for nDCG@10, bpref, and RBP@10 measure, respectively

Table 25 Statistical significance between results of the CLEF2016’s best settings using CLEF2016-2017 validation data in Table 13 (bottom): the high coverage queries set. n, b, and r show statistically significant ($\hbox {pairwise bonferroni} < 0.05$) for nDCG@10, bpref, and RBP@10 measure, respectively

Table 26 Statistical significance between results using condensed evaluation in Table 14. p, m, n, and r show statistically significant ($\hbox {pairwise bonferroni} < 0.05$) for P@10, map, nDCG@10, and RBP@10 measure, respectively

Appendix 2: List of abbreviations

	Abbreviation	Definition
General	CHS	Consumer health search
	CHV	Consumer health vocabulary
	EQFE	Entity query feature expansion
	HT	Health term
	IR	Information retrieval
	KB	Knowledge base
Methods	CC	CHV Construction
	CEM	CHV entity mapping
	CME	CHV mention extraction
	CSE	CHV source of expansion
	EM	Entity mapping
	ME	Mention extraction
	PRF	Pseudo relevance feedback
	PRFHT	Pseudo relevance feedback health term
	RF	Relevance feedback
	RFHT	Relevance feedback health term
	SE	Source of expansion
	UC	UMLS construction
	UEM	UMLS entity mapping
	UME	UMLS mention extraction
	UMLS	Unified medical language system
	USE	UMLS source of expansion
	WC	Wikipedia construction
	WEM	Wikipedia entity mapping
	WME	Wikipedia mention extraction
	WSE	Wikipedia source of expansion
Measures	${<}{\hbox {e,g,l}}{>}$	<Number of expanded queries, queries with gain, queries with loss>
	$\overline{|exp|}$	The average number of terms added in the expanded query
	bpref	Binary preference
	MAP	Mean average precision
	nDCG@10	Normalised discounted cumulative gain at rank 10
	P@10	Precission at rank 10
	RBP@10	Rank-biased precision at rank 10
	Res.	Residual of the rank-biased precision