I found the aeticle in a post on the fediverse, and I can’t find it anymore.
The reaserchers asked a simple mathematical question to an LLM ( like 7+4) and then could see how internally it worked by finding similar paths, but nothing like performing mathematical reasoning, even if the final answer was correct.
Then they asked the LLM to explain how it found the result, what was it’s internal reasoning. The answer was detailed step by step mathematical logic, like a human explaining how to perform an addition.
This showed 2 things:
-
LLM don’t “know” how they work
-
the second answer was a rephrasing of original text used for training that explain how math works, so LLM just used that as an explanation
I think it was a very interesting an meaningful analysis
Can anyone help me find this?
glizzyguzzler@lemmy.blahaj.zone 2 days ago
Can’t help but here’s a rant on people asking LLMs to “explain their reasoning” which is impossible because they can never reason (not meant to be attacking OP, just attacking the “LLMs think and reason” people and companies that spout it):
LLMs are just matrix math to complete the most likely next word. They don’t know anything and can’t reason.
Anything you read or hear about LLMs or “AI” getting “asked questions” or “explain its reasoning” or talking about how they’re “thinking” is just AI propaganda to make you think they’re doing something LLMs literally can’t do but people sure wish they could.
In this case it sounds like people who don’t understand how LLMs work eating that propaganda up and approaching LLMs like there’s something to talk to or discern from.
If you waste egregiously high amounts of gigawatts to put everything that’s ever been typed into matrices you can operate on, you get a facsimile of the human knowledge that went into typing all of that stuff.
It’d be impressive if the environmental toll making the matrices and using them wasn’t critically bad.
TLDR; LLMs can never think or reason, anyone talking about them thinking or reasoning is bullshitting, they utilize almost everything that’s ever been typed to give (occasionally) reasonably useful outputs that are the most basic bitch shit because that’s the most likely next word at the cost of environmental disaster
peoplebeproblems@midwest.social 2 days ago
People don’t understand what “model” means. That’s the unfortunate reality.
adespoton@lemmy.ca 2 days ago
They walk down runways and pose for magazines. Do they reason? Sometimes.
random_character_a@lemmy.world 2 days ago
Yeah. That’s because peoples unfortunate reality is a “model”.
theunknownmuncher@lemmy.world 2 days ago
It’s true that LLMs aren’t “aware” of what internal steps they are taking, so asking an LLM how they reasoned out an answer will just output text that statistically sounds right based on its training set, but to say something like “they can never reason” is provably false.
Its obvious that you have a bias and desperately want reality to confirm it, but there’s been significant research and progress in tracing internals of LLMs, that show logic, planning, and reasoning. Neural networks and very powerful, after all, you are one too. Can you reason?
glizzyguzzler@lemmy.blahaj.zone 2 days ago
Too deep on the AI propaganda there, it’s completing the next word. You can give the LLM base umpteen layers to make complicated connections, still ain’t thinking.
The LLM corpos trying to get nuclear plants to power their gigantic data centers while AAA devs aren’t trying to buy nuclear plants says that’s a straw man and you simultaneously also are wrong.
Using a pre-trained and memory-crushed LLM that can run on a small device won’t take up too much power. But that’s not what you’re thinking of. You’re thinking of the LLM only accessible via ChatGPT’s api that has a yuge context length and massive matrices that needs hilariously large amounts of RAM and compute power to execute. And it’s still a facsimile of thought.
It’s okay they suck and have very niche actual use cases - maybe it’ll get us to something better. But they ain’t gold, they ain’t smart, and they ain’t worth destroying the planet.
ohwhatfollyisman@lemmy.world 2 days ago
would there be a source for such research?
A_Union_of_Kobolds@lemmy.world 2 days ago
AnneBonny@lemmy.dbzer0.com 2 days ago
How would you prove that someone or something is capable of reasoning or thinking?
glizzyguzzler@lemmy.blahaj.zone 2 days ago
You can prove it’s not by doing some matrix multiplication and seeing its matrix multiplication. Much easier way to go about it
WolfLink@sh.itjust.works 1 day ago
The environmental toll doesn’t have to be that bad. You can get decent results from single high-end gaming GPU.
Treczoks@lemmy.world 2 days ago
I’ve read that article. They used something they called an “MRI for AIs”, and checked e.g. how an AI handled math questions, and then asked the AI how it came to that answer, and the pathways actually differed. While the AI talked about using a textbook answer, it actually did a different approach. That’s what I remember of that article.
But yes, it exists, and it is science, not TicTok
lgsp@feddit.it 2 days ago
Thank you. I found the article, linkin the OP
just_another_person@lemmy.world 2 days ago
It’s a developer option that isn’t generally available on consumer-facing products. It’s literally just a debug log that outputs the steps to arrive at a response, nothing more.
It’s not about novel ideation or reasoning (programmatic neural networks don’t do that), but just an output of statistical data that says “Step was 90% certain, Step 2 was 89% certain…etc”
AnneBonny@lemmy.dbzer0.com 2 days ago
Who has claimed that LLMs have the capacity to reason?
theparadox@lemmy.world 2 days ago
More than enough people who claim to know how it works think it might be “evolving” into a sentient being inside it’s little black box. Example from a conversation I gave up on… sh.itjust.works/comment/18759960
adespoton@lemmy.ca 2 days ago
The study being referenced explains in detail why they can’t. So I’d say it’s Anthropic who stated LLMs don’t have the capacity to reason, and that’s what we’re discussing.
The popular media tends to go on and on about conflating AI with AGI and synthetic reasoning.