The U.S. Medical Licensing Exam (USMLE) is considered the gold standard and is known for its difficulty to pass. However, researchers are now reporting that ChatGPT could successfully see to it.
ChatGPT
ChatGPT is a novel artificial intelligence (AI) tool that is used to produce human-like writing. The large language model (LLM), which has been a lot in the news lately and is generating a debate, writes by predicting how words complement one another in a sentence.
The tool supposedly lacks the ability to search the internet, unlike most other chatbots.
You may think of ChatGPT as a much superior version of the predictive text on your phone. A major difference is that it automatically produces text through internal process-driven predictions after being issued a command.
In this new study published in the peer-reviewed journal PLOS Digital Health, researchers report that ChatGPT can score around the 60 percent passing mark for the USMLE. The tool impressed with its logical responses and striking insights.
This is impressive for an examination that is notoriously hard. Many medical students and interns struggle to pass this exam, which requires hundreds of hours to prepare for.
Exceeding performance
The USMLE is a three-step process – it rolls three exams into one. It is highly standardized and covers everything from many things from basic science theories to diagnostic reasoning and bioethics.
In this study, Dr. Tiffany Kung and her AnsibleHealth colleagues put ChatGPT to the test on the USMLE after reportedly first checking to ensure that answers were not available via Google. This means it would only generate answers using the data it was trained with.
The researchers removed questions that were based on images from the June 2022 release of the medical licensing exam. They tested the AI system on 350 out of 376 publicly available questions.
Read Also: De Novo Antibody Design Made Possible by Generative Artificial Intelligence
ChatGPT amazed with a score ranging from 52.4 percent to 75 percent across all three exams, making up the USMLE. The qualifying mark is typically around 60 percent.
Researchers revealed that the tool showed roughly 95 percent concordance across the responses it generated. It also provided one or more significant insights – something that’s “new, non-obvious, and clinically valid – in around 89 percent of its responses.
“Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation,” the researchers said.
ChatGPT also outperformed PubMedGPT, an AI model that was trained wholly on biomedical literature. That other tool scored 50.8 percent on USMLE-style questions from an older dataset.
The innovative AI tool did more than achieve a good score in the USMLE. Researchers stated that it also helped in the writing of their manuscript by synthesizing, simplifying, and providing counterpoints to drafts.
The research team noted that its results suggest that LLMs could aid medical education and clinical decision-making. At AnsibleHealth, clinicians are already using ChatGPT to rewrite technical reports to make them easier for patients to understand.
References
https://www.sciencealert.com/chatgpt-can-almost-pass-the-us-medical-licensing-exam
