AI impresses doctors in answering medical questions

AI impresses doctors in answering medical questions

ChatGPT AI Program Shows Promise in Providing Medical Information

AI Chatbot

The ChatGPT artificial intelligence program, an advanced language model, has the potential to become a reliable source of comprehensive medical information. Although it is not yet perfect, a recent study published in JAMA Network Open indicates that ChatGPT demonstrates accuracy and completeness in answering medical questions.

Embracing the AI Frontier in Healthcare

In today’s digital age, people and doctors frequently rely on search engines like Google and Bing to find answers to their health-related inquiries. With the emergence of AI programs like ChatGPT, the potential for these technologies to revolutionize medical research and provide accurate information appears promising. Dr. Douglas Johnson, the senior researcher and director of the Melanoma Clinical Research Program at Vanderbilt-Ingram Cancer Center, recognized the importance of understanding the accuracy and completeness of AI programs across various medical disciplines.

Study Design and Results

To assess ChatGPT’s capabilities, the researchers enlisted the help of 33 physicians from 17 specialties to create a set of 284 questions with varying difficulty levels. The AI program’s responses were evaluated based on accuracy and completeness.

The study found that ChatGPT provided responses that were mostly to almost completely correct, with an average accuracy score of 4.8 on a 6-point scale. The program demonstrated better accuracy in answering questions related to common conditions, melanoma, and immunotherapy, scoring 5.7, and 5.2, respectively. The program also fared well in answering “yes/no” questions compared to open-ended questions.

Room for Improvement

While ChatGPT’s performance was impressive, it still had some limitations. Occasionally, it provided incorrect or incomplete answers. For example, when asked about oral antibiotics for treating MRSA infections, the response included some options that were not available orally and omitted one of the most important oral antibiotics. However, it’s worth noting that the phrasing of the questions could have influenced these inaccuracies.

The study also revealed that ChatGPT’s accuracy improved when the same question was resubmitted a week or two later. This indicates that the AI program has the ability to learn and adapt over time.

Enhancing Reliability and Usability

To increase both the accuracy and reliability of AI chatbots like ChatGPT, researchers suggest training them on reliable medical information. Incorporating a second AI program to review and verify initial responses also proved beneficial. By continuing to refine and enhance these AI programs, the goal of achieving a higher level of reliability and accuracy in medical information provision becomes more attainable.

AI as a Complementary Tool, Not a Replacement

It is highly unlikely that AI will replace physicians entirely. Instead, it will serve as a helpful tool for both doctors and patients. Doctors can consult AI for additional information when faced with challenging diagnoses, while patients can use the program as a “health coach” for preliminary guidance.

However, concerns have been raised about potential cost-cutting measures in healthcare systems, wherein AI becomes the primary resource for patient advice, potentially leading to a decrease in continuity of care and coordination. Dr. Steven Waldren, Chief Medical Informatics Officer for the American Academy of Family Physicians, emphasized the need to maintain the essential roles of physicians while integrating AI as another valuable resource in the healthcare ecosystem.

Looking Ahead

The future holds promising possibilities for AI chatbots in the medical field. As these technologies continue to improve and become more specialized for medical use, accuracy and reliability are expected to increase. Ultimately, AI chatbots could streamline healthcare processes, enhance access to care, and provide valuable support for both doctors and patients.



ChatGPT, an AI language model, has demonstrated promise in providing accurate and comprehensive medical information. Although not flawless, its ability to learn and adapt over time indicates a positive trajectory for the future of AI in healthcare. While AI chatbots will not replace physicians, they offer valuable assistance, leading to improved patient care and accessibility to medical information. Responsible integration of AI technologies will preserve the essential roles of healthcare professionals and ensure the provision of optimal care.