Using Generative AI as a substitute for professional judgement is a disaster waiting to happen. LLMs aren’t sentient and will frequently hallucinate answers. It’s only a matter of time before incorrect output will lead to catastrophic consequences and the idiot who trusted the LLM, not the LLM itself, will be responsible.
Fears for patient safety as GPs use ChatGPT to diagnose and treat illness
Submitted 1 month ago by thehatfox@lemmy.world to unitedkingdom@feddit.uk
https://inews.co.uk/news/health/patient-safety-fears-chatgpt-gps-diagnosis-3280876
Comments
Mrkawfee@feddit.uk 1 month ago
echodot@feddit.uk 1 month ago
If you read the article that’s not what’s happening here.
Doctors are just using AI like they use any tool. To inform their decision.
Streetlights@lemmy.world 1 month ago
20 years ago there were complaints that GP’s were using Google, now its normal. Can’t help but feel the same will happen here.
TheGrandNagus@lemmy.world 1 month ago
You’re right. Within 10 seconds I just found an article from 2006 saying just that. Earlier ones likely exist.
Swedneck@discuss.tchncs.de 1 month ago
to be fair back then google just showed you what you searched for, i’m not as happy about people googling stuff these days. With AI we already know that it tends to make shit up, and it might very well only get worse as they start being trained on their own output.
echodot@feddit.uk 1 month ago
Actually hallucinations have gone down as AI training has increased. Mostly through things like prompting them to provide evidence. When you prompt them to provide evidence they don’t hallucinate in the first place.
The problem is really to do with the way the older AIs were originally trained. They were basically trained on data where a question was asked, and then a response was given. Nowhere in the data set was there a question that was asked, and the answer was “I’m sorry I do not know”, so the AI basically was unintentionally taught that it is never acceptable to not answer a question. More modern AI have been trained in a better way and have been told it is acceptable not to answer a question. Combined with the fact that they now have the ability to perform internet searches, so like a human they can go look up data if they recognize that they don’t have access to it in their current data set.
That being said, Google’s AI is an idiot.
echodot@feddit.uk 1 month ago
The headline and the article are completely mismatched.
Basically all the article is saying is that doctors sometimes use AI. Which is a bit like saying sometimes doctors look things up in books. Yeah, course they do.
If somebody comes in with a sore throat and the AI prescribes morphine the doctor is probably smart enough to not do that so I don’t really think there’s a major issue here. They are skilled medical professionals they’re not blindly following the AI.
s12@sopuli.xyz 1 month ago
Pretty sad though.
DonPiano@feddit.org 1 month ago
They need to lose their licenses.
Everyone anywhere using one on the job should be fired, but medical personnel is endangering people.
mannycalavera@feddit.uk 1 month ago
The best use of AI at the moment is to act as a tool to quickly search and present data quicker than humanly possible. Not to act upon the findings blindly.
It’s not as easy to say anyone using AI should be fired. There needs to be a more nuanced approach to this. It wholly depends on what the GP did with the information it presented.
An example: back in the day GPs had a huge book of knowledge they would defer to that was peer researched and therefore trusted. If you came in with an odd symptom they’d spend time (often in front of you) flipping through the book to find that elusive disease they read about that one time at university. Later that knowledge moved to a traditional search engine. Why wouldn’t you now use AI to make that search faster? The AI can easily be trained on this same corpus of knowledge.
Of course the GP should double check what they are being told. But simply using AI is not the problem you make it out to be. If you have a corpus of knowledge and the GP uses this in a dangerous way then the GP should be fired. But you don’t then burn the book they found this information from.
thehatfox@lemmy.world 1 month ago
I think the difference here is that medical reference material is based on long process of proven research. It can be trusted as a reliable source of information.
AI tools however are so new they haven’t faced anything like the same level of scrutiny. For now they can’t be considered reliable, and their use should be kept within proper medical trials until we understand them better.
Yes human error will also always be an issue, but putting that on top of the currently shaky foundations of AI only compounds the problem.
TheGrandNagus@lemmy.world 1 month ago
Indeed. GPs have been doing this for a long time. It’s nothing new, and expecting every GP to know every single ailment that humanity has ever experienced, to recall it quickly, and immediately know the course of action to take, is unreasonable.
Like you say, if they’re blindly following a generic ChatGPT instance, then that’s bad.
If they’re aiding their search using an LLM that has been trained on a good medical dataset, then taking that and looking more into it, then there’s no issue.
People have become so reactionary to LLMs and other ‘AI’ stuff.
DonPiano@feddit.org 1 month ago
AI != LLM
YungOnions@sh.itjust.works 1 month ago
‘Everyone anywhere’? That’s an amazingly broad statement. What’re you defining as ‘using one’? If I use ChatGPT to rewrite a paragraph, should I be fired? What about if a non native speaker uses it to remove grammatical errors from an email, should they be fired? How about using it for assisting with coding errors? Or generating draft product marketing copy? Or summarising content for third parties to make it easier to understand? Still a fireable offence? How about generating insights from data? Assistance with Roadmap prioritisation? Generating summaries of meeting notes or presentations? Helping users with learning disabilities understand complex information? Or helping them with letters, emails etc? How about if it use it to remind me of tasks? Or managing my routines?
TheGrandNagus@lemmy.world 1 month ago
Don’t you be bringing nuance into this.
If you used an LLM to find that mistyped variable name, you deserve to lose your job. You and your family must suffer.
If you are blind and you use a screen reader with some AI features, you should be fired. You and your family must suffer.
DonPiano@feddit.org 1 month ago
Yes.
cynar@lemmy.world 1 month ago
It’s depends purely on how it’s used. Used blindly, and yes, it would be a serious issue. It should also not be used as a replacement for doctors.
However, if they could routinely put symptoms into an AI, and have it flag potential conditions, that would be powerful. The doctor would still be needed to sanity check the results and implement things. If it caught rare conditions or early signs of serious ones, that would be a big deal.
AI excels at pattern matching. Letting doctors use it to do that efficiently, to work beyond there current knowledge base is quite a positive use of AI.