The latest artificial intelligence (AI) chatbot, ChatGPT-4, is demonstrating improved accuracy in answering multiple-choice practice questions from OphthoQuestions, a widely-used preparation resource for the ophthalmology board certification.
Marked Improvement in ChatGPT-4’s Performance
A research study conducted by the University of Toronto reveals that the updated version of the chatbot now correctly answers 84% of the practice questions. This result showcases a marked improvement from earlier versions, which answered 46% and 58% correctly in January and February 2023, respectively. The study’s lead investigator, Dr. Rajeev H. Muni, states, “The updated version of this chatbot appears to have improved performance across all question categories on OphthoQuestions.”
The Chatbot’s Role in Ophthalmology Training
This new data builds on existing evidence that AI chatbots can produce human-like responses to prompts, improving conversational AI systems. The older version of ChatGPT successfully answered almost half of the multiple-choice questions from the American Board of Ophthalmology examination. The Toronto-based team tested the updated chatbot on the same Ophthalmic Knowledge Assessment Program (OKAP) and Written Qualifying Exam (WQE) questions used in the previous study.
Analysis of the Chatbot’s Responses
The analysis, performed using Microsoft Excel, showcased the chatbot’s high performance: 84% of 125 text-based multiple-choice questions were answered correctly. The chatbot performed exceptionally well in general medicine, retina and vitreous, and uveitis questions. However, it struggled slightly with clinical optics, answering 62% of these questions correctly.
On average, 71% of ophthalmology trainees selected the same response to the multiple-choice questions as the chatbot, showing its potential as a training resource. However, the study’s authors caution that the chatbot’s performance could vary in official examinations.
Looking Forward
Muni and his team emphasized the importance of interpreting these results within their context. They noted, “the chatbot’s knowledge corpus will likely continue to expand rapidly.” Hence, the team acknowledges the necessity of future studies to assess the chatbot’s evolving performance.
Related Articles
