Submitted 1 year ago by stopthatgirl7@kbin.social to technology@lemmy.world
https://www.axios.com/2023/08/29/chatgpt-medical-diagnosis-study
As AI capabilities advance in complex medical scenarios that doctors face on a daily basis, the technology remains controversial in medical communities.
To allow ChatGPT or comparable AI models to be deployed in hospitals, Succi said that more benchmark research and regulatory guidance is needed, and diagnostic success rates need to rise to between 80% and 90%.
Sucks if your one of the 10-20% who will die because some doctor doesn’t have time to double check. But hey … efficiency!
Or one of the ninty nine percent of people who don’t give the AI their symptoms in medical terminology.
A few things, that’s an abysmal rate when it comes to people’s health. A doctor with that success rate would be sued into next century. The rate dropped further when it came to differential diagnosis, implying chat gpt was leaving out important rarer possibilities. Often doctor’s work by starting with the most common and narrow down from there after repeated rounds of testing if it ends up being something uncommon, but one of their primary jobs is also thinking about rarer dangerous stuff that can mimic more common things and must be ruled out immediately.
Most importantly, the information fed into this was optimized with accurate descriptive medical terminology. This is a language that, in general, patients do not speak. People can also describe things very differently, for instance weak when a doctor may so numb or visa versa. And dizzy could mean just about anything. Someone typing their own story directly into chat gpt is going to get much worse results than this without someone to interpret the word choices and ask the right questions that people may not even realize are important.
Anyways, the possibilities of AI use in Healthcare is interesting, but disappointing it does worse the less common things get and is bad at a differential diagnosis, the areas that would be really helpful as an aid to diagnosis. Some other areas to think about though could be maybe as a front end to find clinical trials with the us gov database, which can be hard to browse, or maybe streamlining the endless insurance paperwork. I’d be surprised insurance companies don’t use something similar already.
. The rate dropped further when it came to differential diagnosis, implying chat gpt was leaving out important rarer possibilities.
Every decade, new diagnoses and discoveries are made. Many manmade or climate change related. We never had microplastic poison before. Or random chemicals added into our foods because it gives a company that +1% profit.
In other words, we are finding new ways to destroy our bodies!
And with AI is always working with historical data, it’ll be a long time before we only use a AI Doctor.
Anecdotally my doctor is prolly only right 75% of the time too. But I’ll go back to him later with more information about the ailment (or more tests) and he’ll eventually get it right. Medical diagnoses in general are not terribly accurate.
I don’t think your comment really disputes anything the previous poster said.
But maybe you read the first lines and skimmed the rest (I certainly do that more often than I would like).
IME my doctor would not outperform ChatGPT
What was the result in the human’s group?
AI could ultimately improve both the efficiency and the accuracy of diagnosis as healthcare in the U.S. gets more expensive and complicated as individuals live longer, and the overall population ages
When you read some bs on internet but thankfully its US only
First in medical spending. 40th or 50th in positive medical outcomes.
Among Western nations.
So it’s about as good as people going to WebMD and diagnosing themselves? hoo boy
That’s a whole 22% better than a coin toss!!!
Not really comparable since a medical diagnosis has hundreds of possible results
Having made many models that were only slightly better than a coin toss, that’s really not bad. Especially since that’s not even their primary design goal.
Details matter here. Here's a few from the study:
ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=–15.8%; P<.001) and clinical management (β=–7.4%; P=.02) question types.
At the time of the study, 36 vignette modules were available on the web, and 34 of the 36 were available on the web as of ChatGPT’s September 2021 training data cutoff date. All 36 modules passed the eligibility criteria of having a primarily textual basis and were included in the ChatGPT model assessment.
All questions requesting the clinician to analyze images were excluded from our study, as ChatGPT is a text-based AI without the ability to interpret visual information.
Those odds are shit. Meanwhile most of us have zero doctor.
Sadly, that percentage is probably better than a good number of doctors.
About 70 percentage points better than my doctor. Nice!
Yeah, medicine is one of the areas where I really feel like AI could make serious strides. Most people don't have a doctor they see regularly anyway so any input would be welcome. Anecdotally I've known several people who were misdiagnosed or just had doctors not believe them.
Of course I'd want to be able to escalate and have different treatment options but I could probably be ok with AI-assisted medicine.
Medicine is going to take awhile for anything except small-scope tools to handle one specific thing, due to the massive variation in the presentation of different problems that doctors can face.
What won’t take as long, because there isn’t the same inherent variation in presentation, is law.
I don’t use ChatGPT and ain’t planning to, but someone should try asking it something like…
“How often should a male change their tampon?”
See what, if any nonsense it regurgitates.
“Men do not typically use tampons since they are designed for menstruation, which is a female biological process. If you have specific questions about personal hygiene or healthcare, it’s best to consult with a medical professional who can provide guidance based on your individual needs and circumstances.”
Okay then, well go figure. I was guessing it would puke up some nonsense, but apparently not.
Men that menstruate exist.
Now I’m no expert, but if you’re bleeding from your penis or your anus, you should probably go see a doctor about that.
CyberCatBytes@kbin.social 1 year ago
I mean if the AI takes women seriously then that's honestly already better than most of the doctors I've had
hydroel@lemmy.world 1 year ago
Unfortunately, if the data is biased, the model is biased.
Contramuffin@lemmy.world 1 year ago
Yes, that’s something that’s constantly emphasized in scientific research. You might have the most infallible algorithm, but… garbage in, garbage out. You’ll still get garbage data if what you enter into the algorithm is garbage
Domille@sh.itjust.works 1 year ago
I was about to say…
Wonder what the success rate of doctors is. I’d be surprised if it is above 70% lol
stopthatgirl7@kbin.social 1 year ago
Or if it’s better at diagnosing minorities, too.
cooopsspace@infosec.pub 1 year ago
It almost certainly won’t, but it’s nice to hope.
It might remove the face to face human bias of a GP but it doesn’t make up for the decades of preconceived or absent research about women or minorities.