Extracting Decision Trees from Medical Texts: An Overview of the Text2DT Track in CHIP2022

Abstract

This paper presents an overview of the Text2DT shared task$^{1}$ held in the CHIP-2022 shared tasks. The shared task addresses the challenging topic of automatically extracting the medical decision trees from the un-structured medical texts such as medical guidelines and textbooks. Many teams from both industry and academia participated in the shared tasks, and the top teams achieved amazing test results. This paper describes the tasks, the datasets, evaluation metrics, and the top systems for both tasks. Finally, the paper summarizes the techniques and results of the evaluation of the various approaches explored by the participating teams.$^{1}$(http://cips-chip.org.cn/2022/eval3)

A Manual Evaluation of Annotated MDTs

The detail of our manual evaluation of medical decision trees are as follows:

• 1. We observed the subjects’ performance on medical decision problems of similar difficulty under medical texts and MDTs. Specifically, subjects will answer three sets of medical decision questions, each group providing texts or decision trees containing the medical knowledge needed to answer the medical decision question. We observe their accuracy and time spent answering the decision question. Each set of questions is randomly selected from the question pool and is guaranteed to be of similar difficulty.

• 2. We invited subjects to rate medical texts and MDTs in terms of readability, completeness, and helpfulness. Specifically, we randomly selected five medical texts and MDTs expressing the same knowledge. We asked subjects to score (0–3) them in terms of whether they were clear and easy to understand (readability), whether they were comprehensive and detailed (completeness), and whether they were helpful in understanding or studying medical knowledge (helpfulness).

Table 6. Results of manual evaluation of annotated MDTs. The results in the first field are for subjects without medical background, and the results in the second field are for medical practitioners. A represents the average accuracy of answering the medical decision questions. T represents the average seconds spent answering the medical decision questions. R, C, and H represent the readability, completeness, and helpfulness average scores.

The results of the manual evaluation are shown in Table 6. We can draw the following conclusions:

For subjects without medical background, the medical decision tree helped them make more correct decisions in less time compared with the medical text and gained the highest scores for readability, completeness, and helpfulness. Theoretically, the completeness of the medical text should be better than the medical decision tree. Still, due to the poor readability of the medical text, the subjects may not have gained complete access to the knowledge contained in the medical text.

For medical practitioners, the medical decision tree group achieved the same accuracy on the medical decision questions as the medical text group, but the former took less time. The medical decision trees gained the highest readability and helpfulness scores and slightly lower completeness than the medical texts. The results demonstrate that the medical decision tree can help people make treatment decisions faster and better and can model medical decision knowledge clearly and intuitively, which can help readers better understand medical decision knowledge.