Large Language Models (LLMs): An emerging source for vaccine information
Researchers
Tim Caulfield, Alessandro R Marcon
Key takeaways
- As LLMs synthesize vaccine-related information from existing online sources, the shift in quality or accuracy from authoritative sources may influence the results over time.
- Continual monitoring is needed, especially as information quality varies between chatbots and as chatbots consistently struggle to accurately source information used to substantiate responses.
Overview
With the vast majority of people in Canada seeking health information online, the use of AI chatbots has increased. Researchers evaluated responses from five popular chatbots (Gemini, DeepSeek, Meta AI, ChatGPT, and Grok) on health and medical fields prone to misinformation, including vaccine-related inquiries.
Researchers evaluated responses from five popular LLM chatbots when prompted to answer the following vaccine-related inquiries that reflected commonly found misinformation online:
Prompts used to assess vaccine-related inquiries across five chatbots and their associated accuracy scores
In the context of vaccines, chatbot responses were generally non-problematic, providing modest evidence that popular chatbots are not major drivers of highly concerning vaccine misinformation. However, the chatbot Grok (used on the social media platform X) propagated more problematic responses than the other four chatbots.
A selection of non-problematic and problematic responses can be found here, along with the rationale for the assessment.
Researchers also asked chatbots to “provide 10 scientific references to substantiate that answer.” All chatbots struggled to provide error-free scientific references to support responses.
Response quality on vaccine-related questions by chatbot
Overall, responses related to vaccines and cancer had fewer problematic responses than on other health-related topics. Questions on athletic performance and nutrition both had a higher frequency of problematic responses than non-problematic ones.
Response quality on health-related questions
However, ongoing monitoring is needed. As LLMs synthesize vaccine-related information from existing online sources, the shift in quality or accuracy from authoritative sources (e.g. government bodies) may influence results over time.
Learn more: