Assessing ChatGPT's Performance in Health Fact-Checking: Performance, Biases, and Risks

Abstract

The increasing use of ChatGPT by the general public has prompted us to assess ChatGPT's performance in health fact-checking and uncover potential biases and risks arising from its utilization. In this study, we employed two publicly accessible datasets to evaluate ChatGPT's performance. We utilized BERTopic for clustering health claims into topics and subsequently employed the gpt-3.5-turbo API for fact-checking these claims. ChatGPT's performance was appraised on multi-class (False, Mixture, Mostly-False, Mostly-True, True) and binary (True, False) levels, with a thorough analysis of its performance across various topics. ChatGPT achieved a F1-score of 0.54 and 0.64 in the multi-class task and 0.88 and 0.85 in the binary task on the two datasets, respectively. In most health topics (e.g., vaccines, Covid-19), ChatGPT's F1-score exceeded 0.8, except for specific topics, such as novel or contentious cancer treatments, which yielded a F1-score below 0.6. We scrutinized the erroneous fact-checking labels and explanations provided by ChatGPT, revealing that it may produce inaccurate results for claims with misleading intent, inaccurate information, emerging research findings, or contentious health knowledge.