Large language models aren’t designed to be knowledge machines - they’re designed to generate natural-sounding language, nothing more. The fact that they ever get things right is just a byproduct of their training data containing a lot of correct information. These systems aren’t generally intelligent, and people need to stop treating them as if they are. Complaining that an LLM gives out wrong information isn’t a failure of the model itself - it’s a mismatch of expectations.
AI Chatbots Remain Overconfident — Even When They’re Wrong: Large Language Models appear to be unaware of their own mistakes, prompting concerns about common uses for AI chatbots.
Submitted 1 day ago by Pro@programming.dev to technology@lemmy.world
https://www.cmu.edu/dietrich/news/news-stories/2025/july/trent-cash-ai-overconfidence.html
Comments
Perspectivist@feddit.uk 1 day ago
shalafi@lemmy.world 1 day ago
Neither are our brains.
“Brains are survival engines, not truth detectors. If self-deception promotes fitness, the brain lies. Stops noticing—irrelevant things. Truth never matters. Only fitness. By now you don’t experience the world as it exists at all. You experience a simulation built from assumptions. Shortcuts. Lies. Whole species is agnosiac by default.”
― Peter Watts, Blindsight (fiction)
Starting to think we’re really not much smarter. “But LLMs tell us what we want to hear!” Been on FaceBook lately, or lemmy?
If nothing else, LLMs have woke me to how stupid humans are vs. the machines.
Perspectivist@feddit.uk 1 day ago
There are plenty of similarities in the output of both the human brain and LLMs, but overall they’re very different. Unlike LLMs, the human brain is generally intelligent - it can adapt to a huge variety of cognitive tasks. LLMs, on the other hand, can only do one thing: generate language. It’s tempting to anthropomorphize systems like ChatGPT because of how competent they seem, but there’s no actual thinking going on. It’s just generating language based on patterns and probabilities.
aesthelete@lemmy.world 1 day ago
Every thread about LLMs has to have some guy like yourself saying how LLMs are like humans and smarter than humans for some reason.
jj4211@lemmy.world 1 day ago
It’s not that they may be deceived, it’s that they have no concept of what truth or fiction, mistake or success even are.
Our brains know the concepts and may fall to deceipt without recognizing it, but we at least recognize that the concept exists.
An AI generates content that is a blend of material from the training material consistent with extending the given prompt. It only seems to introduce a concept of lying or mistakes when the human injects that into the human half of the prompt material. It will also do so in a way that the human can just as easily instruct it to correct a genuine mistake as well as the human instruct it to correct something that is already correct (unless the training data includes a lot of reaffirmation of the material in the face of such doubts).
An LLM can consume more input than a human can gather in multiple lifetimes and still bo wonky in generating content, because it needs enough to credibly blend content to extend every conceivable input. It’s why so many people used to judging human content get derailed by judging AI content. An AI generates a fantastic answer to an interview question that only solid humans get right, only to falter ‘on the job’ because the utterly generic interview question looks like millions of samples in the input, but the actual job was niche.
kureta@lemmy.ml 16 hours ago
People should understand that words not “unaware” or “overconfident” are not even applicable to these pieces of software. We might build intelligent machines in the future but if you know how these large language models work, it is obvious that it doesn’t even make sense to talk about the awareness, intelligence, or confidence of such systems.
turmacar@lemmy.world 10 hours ago
I find it so incredibly frustrating that we’ve gotten to the point where the “marketing guys” are not only in charge, but are believed without question, that what they say is true until proven otherwise.
“AI” becoming the colloquial term for LLMs and them being treated as a flawed intelligence instead of interesting generative constructs is purely in service of people selling them as such. And it’s maddening. Because they’re worthless for that purpose.
Baggie@lemmy.zip 20 hours ago
Oh god I just figured it out.
It was never they are good at their tasks, faster, or more money efficient.
They are just confident to stupid people.
Christ, it’s exactly the same failing upwards that produced the c suite. They’ve just automated the process.
SnotFlickerman@lemmy.blahaj.zone 19 hours ago
Oh good, so that means we can just replace the C-suite with LLMs then, right? Right?
rc__buggy@sh.itjust.works 1 day ago
However, when the participants and LLMs were asked retroactively how well they thought they did, only the humans appeared able to adjust expectations
This is what everyone with a fucking clue has been saying for the past 5, 6? years these stupid fucking chatbots have been around.
Modern_medicine_isnt@lemmy.world 1 day ago
It’s easy, just ask the AI “are you sure”? Until it stops changing it’s answer.
But seriously, LLMs are just advanced autocomplete.
Lfrith@lemmy.ca 1 day ago
They can even get math wrong. Which surprised me. Had to tell it the answer is wrong for them to recalculate and then get the correct answer. It was simple percentages of a list of numbers I had asked.
GissaMittJobb@lemmy.ml 1 day ago
Language models are unsuitable for math problems broadly speaking. We already have good technology solutions for that category of problems. Luckily, you can combine the two - prompt the model to write a program that solves your math problem, then execute it. You’re likely to see a lot more success using this approach.
jj4211@lemmy.world 1 day ago
Fun thing, when it gets the answer right, tell it is was wrong and then see it apologize and “correct” itself to give the wrong answer.
saimen@feddit.org 1 day ago
I once gave some kind of math problem (how to break down a certain amount of money into bills) and the llm wrote a python script for it, ran it and thus gave me the correct answer. Kind of clever really.
cley_faye@lemmy.world 1 day ago
Ah, the monte-carlo approach to truth.
jj4211@lemmy.world 1 day ago
I kid you not, early on (mid 2023) some guy mentioned using ChatGPT for his work and not even checking the output (he was in some sort of non-techie field that was still in the wheelhouse of text generation). I expresssed that LLMs can include some glaring mistakes and he said he fixed it by always including in his prompt “Do not hallucinate content and verify all data is actually correct.”.
Passerby6497@lemmy.world 1 day ago
Ah, well then, if he tells the bot to not hallucinate and validate output there’s no reason to not trust the output. After all, you told the bot not to, and we all know that self regulation works without issue all of the time.
jj4211@lemmy.world 1 day ago
They are not only unaware of their own mistakes, they are unaware of their successes. They are generating content that is, per their training corpus, consistent with the input. This gets eerie, and the ‘uncanny valley’ of the mistakes are all the more striking, but they are just generating content without concept of ‘mistake’ or’ ‘success’ or the content being a model for something else and not just being a blend of stuff from the training data.
For example: Me: Generate an image of a frog on a lilypad. LLM: I’ll try to create that — a peaceful frog on a lilypad in a serene pond scene. The image will appear shortly below.
<includes a perfectly credible picture of a frog on a lilypad, request successfully processed>
Me (lying): That seems to have produced a frog under a lilypad instead of on top. LLM: Thanks for pointing that out! I’m generating a corrected version now with the frog clearly sitting on top of the lilypad. It’ll appear below shortly.
<includes another perfectly credible picture>
It didn’t know anything about the picture, it just took the input at it’s word. A human would have stopped to say “uhh… what do you mean, the lilypad is on water and frog is on top of that?” Or if the human were really trying to just do the request without clarification, they might have tried to think “maybe he wanted it from the perspective of a fish, and he wanted the frog underwater?”.
But tha training data isn’t predominantly people blatantly lying about such obvious things or second guessing things that were done so obviously normally correct.
vithigar@lemmy.ca 21 hours ago
The use of language like “unaware” when people are discussion LLMs drives me crazy. LLMs aren’t “aware” of anything. They do not have a capacity for awareness in the first place.
People need to stop taking about them using terms that imply thought or consciousness, because it subtly feeds into the idea that they are capable of such.
LainTrain@lemmy.dbzer0.com 17 hours ago
Okay fine, the LLM does not take into account in the context of its prompt that yada yada. Happy now word police, or do I need to pay a fine too? The real problem is people are replacing their brains with chatbots owned by the rich but go off pat yourself on the back.
fodor@lemmy.zip 1 day ago
What a terrible headline. Self-aware? Really?
cley_faye@lemmy.world 1 day ago
prompting concerns
Oh you.
Lodespawn@aussie.zone 1 day ago
Why is a researcher with a PhD in social sciences researching the accuracy confidence of predictive text, how has this person gotten to where they are without being able to understand that LLM don’t think? Surely they came up when he started even co soldering this brainfart of a research project?
rc__buggy@sh.itjust.works 1 day ago
Someone has to prove it wrong before it’s actually wrong. Maybe they set out to discredit the bots
Lodespawn@aussie.zone 1 day ago
I guess, but it’s like proving your phones predictive text has confidence in its suggestions regardless of accuracy. Confidence is not an attribute of a math function, they are attributing intelligence to a predictive model.
BeMoreCareful@lemmy.world 20 hours ago
There goes middle management
melsaskca@lemmy.ca 1 day ago
If you don’t know you are wrong, when you have been shown to be wrong, you are not intelligent. So A.I. has become “Adequate Intelligence”.
MonkderVierte@lemmy.zip 1 day ago
That definition seems a bit shaky. Trump & co. are mentally sick but they do have a minimum of intelligence.
jol@discuss.tchncs.de 22 hours ago
As any modern computer system, LLMs are much better and smarter than us at certain tasks while terrible at others. You could say that having good memory and communication skills is part of what defines an intelligent person. Not everyone has those abilities, but LLMs do.
My point is, there’s nothing useful coming out of the arguments over the semantics of the word “intelligence”.
CosmoNova@lemmy.world 1 day ago
Is that a recycled piece from 2023? Because we already knew that.
SGGeorwell@lemmy.world 1 day ago
[deleted]Whitebrow@lemmy.world 1 day ago
Not even a good use case either, especially when it spews such bullshit like “there’s no recorded instance of trump ever having used the word enigma” and “there’s 1 r in strawberry”.
LLMs are a copy paste machine, not a rationalization engine of any sort (at least as far as all the slop that we get shoved in our face, I don’t include the specialized protein folding and reconstructive models that were purpose built for very niche applications)
Quill7513@slrpnk.net 1 day ago
they’re solid starting point for shopping now that wirecutter, slant, and others are enshittified. i hate it and it makes me feel dirty to use, and you can’t just do whatever the llm says. but asking it for a list of options to then explore is currently the best way i’ve found to jump into things like outdoor basketball shoe options
etherphon@lemmy.world 1 day ago
Sounds pretty human to me. /s
shalafi@lemmy.world 1 day ago
Sounds pretty human to me. no /s
El_guapazo@lemmy.world 1 day ago
AI evolved their own form of the Dunning Kruger effect.
RoadTrain@lemdro.id 1 day ago
About halfway through the article they quote a paper from 2023:
Similarly, another study from 2023 found LLMs “hallucinated,” or produced incorrect information, in 69 to 88 percent of legal queries.
The LLM space has been changing very quickly over the past few years. Yes, LLMs today still “hallucinate”, but you’re not doing anyone a service by reporting in 2025 the state of the field over 2 years before.
CeeBee_Eh@lemmy.world 1 day ago
This happened to me the other day with Jippity. It outright lied to me:
“You’re absolutely right. Although I don’t have access to the earlier parts of the conversation”.
So it says that I was right in a particular statement, but didn’t actually know what I said. So I said to it, you just lied. It kept saying variations of:
“I didn’t lie intentionally”
“I understand why it seems that way”
“I wasn’t misleading you”
etc
It flat out lied and tried to gaslight me into thinking I was in the wrong for taking that way.
greygore@lemmy.world 1 day ago
It didn’t lie to you or gaslight you because those are things that a person with agency does. Someone who lies to you makes a decision to deceive you for whatever reason they have. Someone who gaslights you makes a decision to behave like the truth as you know it is wrong in order to discombobulate you and make you question your reality.
The only thing close to a decision that LLMs make is: what text can I generate that statistically looks similar to all the other text that I’ve been given. The only reason they answer questions is because in the training data they’ve been provided, questions are usually followed by answers.
It’s not apologizing you to, it knows from its training data that sometimes accusations are followed by language that we interpret as an apology, and sometimes by language that we interpret as pushing back. It regurgitates these apologies without understanding anything, which is why they seem incredibly insincere - it has no ability to be sincere because it doesn’t have any thoughts.
There is no thinking. There are no decisions. The more we anthropomorphize these statistical text generators, ascribing thoughts and feelings and decision making to them, the less we collectively understand what they are, and the more we fall into the trap of these AI marketers about how close we are to truly thinking machines.
CeeBee_Eh@lemmy.world 21 hours ago
The only thing close to a decision that LLMs make is
That’s not true. An “if statement” is literally a decision tree.
The only reason they answer questions is because in the training data they’ve been provided
This is technically true for something like GPT-1. But it hasn’t been true for the models trained in the last few years.
it knows from its training data that sometimes accusations are followed by language that we interpret as an apology, and sometimes by language that we interpret as pushing back. It regurgitates these apologies without understanding anything, which is why they seem incredibly insincere
It has a large amount of system prompts that alter default behaviour in certain situations. Such as not giving the answer on how to make a bomb. I’m fairly certain there are catches in place to not be overly apologetic to minimize any reputation harm and to reduce potential “liability” issues.
And in that scenario, yes I’m being gaslite because a human told it to.
There is no thinking
Partially agree. There’s no “thinking” in sentient or sapient sense. But there is thinking in the academic/literal definition sense.
There are no decisions
Absolutely false. The entire neural network is billions upon billions of decision trees.
The more we anthropomorphize these statistical text generators, ascribing thoughts and feelings and decision making to them, the less we collectively understand what they are
I promise you I know very well what LLMs and other AI systems are. They aren’t alive, they do not have human or sapient level of intelligence, and they don’t feel.
But “gaslighting” is a perfectly fine description of what I explained. The initial conditions were the same and the end result (me knowing the truth and getting irritated about it) were also the same.
kameecoding@lemmy.world 1 day ago
Oh shit, they do behave like humans after all.
Etterra@discuss.online 1 day ago
Confidently incorrect.
SnotFlickerman@lemmy.blahaj.zone 1 day ago
That’s because they aren’t “aware” of anything.
nymnympseudonym@lemmy.world 1 day ago
This Nobel Prize winner and subject matter expert takes the opposite view
youtube.com/watch?v=IkdziSLYzHw&t=2730s
SnotFlickerman@lemmy.blahaj.zone 1 day ago
Interesting talk but the number of times he completely dismisses the entire field of linguists kind of makes me think he’s being disingenuous about his familiarity with it.
For one, I think he is dismissing holotes, the concept of “wholeness.” That when you cut something apart to it’s individual parts, you lose something about the bigger picture. This deconstruction of language misses the larger picture of the human body as a whole, and how every part of us, from our assemblage of organs down to our DNA, impact how we interact with and understand the world. He may have a great definition of understanding but it still sounds (to me) like it’s potentially missing aspects of human/animal biologically based understanding.
For example, I have cancer, and about six months before I was diagnosed, I had begun to get more chronically depressed than usual. I felt hopeless and I didn’t know why. Surprisingly, that’s actually a symptom of my cancer. What understanding did I have that changed how I felt inside and how I understood the things around me? Suddenly I felt different about words and ideas, but nothing had changed externally, something had change internally. The connections in my neural network had adjusted, the feelings and associations with words and ideas was different, but I hadn’t done anything to make that adjustment. No learning or understanding had happened. I had a mutation in my DNA that made that adjustment for me.
Further, I think he’s deeply misunderstanding (possibly intentionally?) what linguists like Chomsky are saying when they say humans are born with language. They mean that we are born with a genetic blueprint to understand language. Just like animals are born with a genetic blueprint to do things they were never trained to do. Many animals are born and almost immediately stand up to walk. This is the same principle. There are innate biologically ingrained understandings that help us along the path to understanding.
Anyway, interesting talk, but I immediately am skeptical of anyone who wholly dismisses an entire field of thought so casually.
greygore@lemmy.world 21 hours ago
I watched this entire video just so that I could have an informed opinion. First off, this feels like two very separate talks:
The first part is a decent breakdown of how artificial neural networks process information and store relational data about that information in a vast matrix of numerical weights that can later be used to perform some task. In the case of computer vision, those weights can be used to recognize objects in a picture or video streams, such as whether something is a hotdog or not.
As a side note, if you look up Hinton’s 2024 Nobel Peace Prize in Physics, you’ll see that he won based on his work on the foundations of these neural networks and specifically, their training. He’s definitely an expert on the nuts and bolts about how neural networks work and how to train them.
He then goes into linguistics and how language can be encoded in these neural networks, which is how large language models (LLMs) work… by breaking down words and phrases into tokens and then using the weights in these neural networks to encode how these words relate to each other. These connections are later used to generate other text output related to the text that is used as input. So far so good.
At that point he points out these foundational building blocks have been used to lead to where we are now, at least in a very general sense. He then has what I consider the pivotal slide of the entire talk, labeled Large Language Models, which you can see at 17:22. In particular he has two questions at the bottom of the slide that are most relevant:
The problem is: he never answers these questions. He immediately moves on to his own theory about how language works using an analogy to LEGO bricks, and then completely disregards the work of linguists in understanding language, because what do those idiots know?
At this point he brings up The long term existential threat and I would argue the rest of this talk is now science fiction, because it presupposes that understanding the relationship between words is all that is necessary for AI to become superintelligent and therefore a threat to all of us.
Which goes back to the original problem in my opinion: LLMs are text generation machines. They use neural networks encoded as a matrix of weights that can be used to predict long strings of text based on other text. That’s it. You input some text, and it outputs other text based on that original text.
We know that different parts of the brain have different responsibilities. Some parts are used to generate language, other parts store memories, still other parts are used to make our bodies move or regulate autonomous processes like our heartbeat and blood pressure. Still other bits are used to process images from our eyes and other parts reason about spacial awareness, while others engage in emotional regulation and processing.
Saying that having a model for language means that we’ve built an artificial brain is like saying that because I built a round shape called a wheel means that I invented the modern automobile. It’s a small part of a larger whole, and although neural networks can be used to solve some very difficult problems, they’re only a specific tool that can be used to solve very specific tasks.
Although Geoffrey Hinton is an incredibly smart man who mathematically understands neural networks far better than I ever will, extrapolating that knowledge out to believing that a large language model has any kind of awareness or actual intelligence is absurd. It’s the underpants gnome economic theory, but instead of:
It looks more like:
If LLMs were true artificial intelligence, then they would be learning at an increasing rate as we give them more capacity, leading to the singularity as their intelligence reaches hockey stick exponential growth. Instead, we’ve been throwing a growing amount resources at these LLMs for increasingly smaller returns. We’ve thrown a few extra tricks into the mix, like “reasoning”, but beyond that, I believe it’s clear that we’re headed towards a local maximum that is far enough away from intelligence that would be truly useful (and represent an actual existential threat), but in actuality only resembles what a human can output well enough to fool human decision makers into trusting them to solve problems that they are incapable of solving.
obinice@lemmy.world 1 day ago
People really do not like seeing opposing viewpoints, eh? There’s disagreeing, and then there’s downvoting to oblivion without even engaging in a discussion, haha.
Even if they’re probably right, in such murky uncertain waters where we’re not experts, one should have at least a little open mind, or live and let live.