Could be better, but still a huge step up from the hate rhetoric magats get spoon fed 24/7 from Fox and friends.
Largest study of its kind shows AI assistants misrepresent news content 45% of the time – regardless of language or territory
Submitted 5 months ago by sqgl@sh.itjust.works to technology@lemmy.world
https://www.bbc.co.uk/mediacentre/2025/new-ebu-research-ai-assistants-news-content
Comments
sin_free_for_00_days@sopuli.xyz 5 months ago
oplkill@lemmy.world 5 months ago
Replace CEOs by AI
morrowind@lemmy.ml 5 months ago
“misrepresent” is a vague term. Actual graph from the study
The main issue is usual… sources. AI is bad at sources without a proper pipeline. They note that Gemini is the worst at 72%.
Note, they’re not testing models with their own pipeline. They’re testing other people’s products. This is more indicative of the product design than the actual models
davidagain@lemmy.world 5 months ago
This graph clearly shows that AI is also shockingly bad at factual accuracy and at telling a news story in such a way that someone who didn’t already know about it to understand the issues and context. I think you’re misrepresenting this graph as being bad about sources, but here’s a better summary of the point you seem to be making:
AI’s summaries don’t match their source data.
So actually, the headline is pretty accurate in calling it misrepresentation.
NotMyOldRedditName@lemmy.world 5 months ago
I’ve had someone else’s AI summarize some content I created elsewhere, and it got it incredibly wrong to the point of changing the entire meaning of my original content.
danc4498@lemmy.world 5 months ago
Makes sense. I have used AI for software development tasks such as manipulating SQL queries and XML files (tedious things) and am always disappointed with how AI will misinterpret some things. But it’s obvious with those when the requests fail. But for things like “the news” where there is no QA team to point out the defect, it will be much harder to notice. And when AI starts (or continues) to use AI generated posts as sources, it will get much worse.
Yerbouti@sh.itjust.works 5 months ago
I dont understand the use people make of AI. I know a lot of of professionnal composer who are like “That’s awesome, AI does the music for me now!” and I’m like, cool, now you only have the boring part of the job to do since the fun part was made by AI. Creating the music is litteraly the only fun part, I hate everything around it.
balsoft@lemmy.ml 5 months ago
It’s a word predictor. It is good at simple text processing. Think local code refactoring, changing the style or structure of a small text piece, or summarizing small text pieces into even smaller text pieces. It is ok at synthesizing new text that has similar structure to the training corpus. Think generating repetitive boilerplate or copywriting. It is very bad at recalling or checking facts, logic, mathematics, and everything else that people seem to be using it for nowadays.
Korkki@lemmy.ml 5 months ago
The info-sphere today is already a highly delusional place and news can be often contradictory, even from day to day especially by outlets like BCC who is more focused on setting global narratives, not being a reporter of facts as best understood at the moment. No wonder AI would be confused, most readers are confused when navigating every statement made by experts or anonymous officials on every subject. Seems like this study really measured an AI models ability to vomit out the same text in different words and avoiding using any outside context be it accurate or hallucination.
floofloof@lemmy.ca 5 months ago
BCC
AnUnusualRelic@lemmy.world 5 months ago
Yet the LLM seems to be what everyone is pushing, because it will supposedly get better. Haven’t we reached the limits of this model and shouldn’t other types of engines be tried?
floofloof@lemmy.ca 5 months ago
shouldn’t other types of engines be tried?
Sure, but the tricky bit is to be more specific than that.
AnUnusualRelic@lemmy.world 5 months ago
Well, you know…
“Waves vaguely”
Jhex@lemmy.world 5 months ago
buT AI iS hERe tO StAY
SaraTonin@lemmy.world 5 months ago
There’s a few replies talking about humans misrepresenting the news. This is true, but part of the problem here is that most people understand the concept of bias - even if only to the extent of “my people neutral, your people biased”. But this is less true for LLMs. There’s research which shows that because LLMs present information authoritatively that not only do people tend to trust them, but they’re actually less likely to check the sources that the LLM provides than they would be with other forms of being presented with information.
And it’s not just news. I’ve seen people seriously argue that fringe pseudo-science is correct because they fed a very leading prompt into a chatbot and got exactly the answer they were looking for.
Axolotl_cpp@feddit.it 5 months ago
I hear a lot of people say “let’s ask chatGPT” like the AI is god and know everthing 🙏, that’s a big problem to be honest
Best_Jeanist@discuss.online 5 months ago
I wonder if people trust ChatGPT more or less than an international celebrity who is also their best friend.
Kissaki@feddit.org 5 months ago
Will they change their disclaimer now, from “can be wrong” to “is often wrong”? /s
moistclump@lemmy.world 5 months ago
And then I wonder how frequently humans misinterpret the mistranslated news.
snooggums@piefed.world 5 months ago
Humans do it often, but they don't have billions of dollars funding their responses.
Treczoks@lemmy.world 5 months ago
Worse: One third of adult actually believe the shit the AI produces.
MonkderVierte@lemmy.zip 5 months ago
Parrot is wrong almost half of the time. Who knew?
altphoto@lemmy.today 5 months ago
Do you realize what you just said!!!
Wow! They have reached parrot intelligence!
Next they might teach it to butterfly! You know, like you’re off the ground and going somewhere in open air, but they just keep building shit right where you’re flying… And lamps!
From there, who knows?!
jordanlund@lemmy.world 5 months ago
I wish they had broke it out by AI. The article states:
“Gemini performed worst with significant issues in 76% of responses, more than double the other assistants, largely due to its poor sourcing performance.”
But I don’t see that anywhere in the linked PDF of the “full results”.
This sort of study should also be re-done from time to time to track AI version numbers.
nick@campfyre.nickwebster.dev 5 months ago
And also which version of the models. Gemini 2.5 Flash is a completely different experience to 2.5 Pro.
Rothe@piefed.social 5 months ago
It doesn’t really matter, “AI” is being asked to do a task it was never meant to do. It isn’t good at it, and it will never be good at it.
Cocodapuf@lemmy.world 5 months ago
Wow, way to completely ignore the content of the comment you’re replying to. Clearly, some are better than others… so, how do the others perform? It’s worth knowing before we make assertions.
The excerpt they quoted said:
“Gemini performed worst with significant issues in 76% of responses, more than double the other assistants, largely due to its poor sourcing performance.”
So that implies that “the other assistants” performed more than twice as well, so presumably that means encountering serious issues less than 38% of the time (still not great, but better). But they said “more than double the other assistants”, does that mean double the rate of one of the others or double the average of the others? If it’s an average it would mean that some models probably performed better, while others performed worse.
This was the point, what was reported was insufficient information.
snooggums@piefed.world 5 months ago
Using an LLM to return accurate information is like using a shoe to hammer a nail.
sirico@feddit.uk 5 months ago
So less of a percentage than the readers and mass media
paraphrand@lemmy.world 5 months ago
Precision, nuance, and up to the moment contextual understanding are all missing from the “intelligence.”
Treczoks@lemmy.world 5 months ago
Like the average American with an 8th grade reading comprehension.
snooggums@piefed.world 5 months ago
Which is what they used for the training data.
FaceDeer@fedia.io 5 months ago
So it's about on par with humans, then.
wasabi@lemmy.world 5 months ago
[deleted]greenbit@lemmy.zip 5 months ago
The fascist social media influencers are already pushing generated bodycam and surveillance videos for xenophobia etc. A large enough mass of the population doesn’t know what’s real and that’s the goal
HugeNerd@lemmy.ca 5 months ago
wrinkle: AI used for this study