I found the aeticle in a post on the fediverse, and I can’t find it anymore.
The reaserchers asked a simple mathematical question to an LLM ( like 7+4) and then could see how internally it worked by finding similar paths, but nothing like performing mathematical reasoning, even if the final answer was correct.
Then they asked the LLM to explain how it found the result, what was it’s internal reasoning. The answer was detailed step by step mathematical logic, like a human explaining how to perform an addition.
This showed 2 things:
-
LLM don’t “know” how they work
-
the second answer was a rephrasing of original text used for training that explain how math works, so LLM just used that as an explanation
I think it was a very interesting an meaningful analysis
Can anyone help me find this?
glizzyguzzler@lemmy.blahaj.zone 3 weeks ago
Can’t help but here’s a rant on people asking LLMs to “explain their reasoning” which is impossible because they can never reason (not meant to be attacking OP, just attacking the “LLMs think and reason” people and companies that spout it):
LLMs are just matrix math to complete the most likely next word. They don’t know anything and can’t reason.
Anything you read or hear about LLMs or “AI” getting “asked questions” or “explain its reasoning” or talking about how they’re “thinking” is just AI propaganda to make you think they’re doing something LLMs literally can’t do but people sure wish they could.
In this case it sounds like people who don’t understand how LLMs work eating that propaganda up and approaching LLMs like there’s something to talk to or discern from.
If you waste egregiously high amounts of gigawatts to put everything that’s ever been typed into matrices you can operate on, you get a facsimile of the human knowledge that went into typing all of that stuff.
It’d be impressive if the environmental toll making the matrices and using them wasn’t critically bad.
TLDR; LLMs can never think or reason, anyone talking about them thinking or reasoning is bullshitting, they utilize almost everything that’s ever been typed to give (occasionally) reasonably useful outputs that are the most basic bitch shit because that’s the most likely next word at the cost of environmental disaster
peoplebeproblems@midwest.social 3 weeks ago
People don’t understand what “model” means. That’s the unfortunate reality.
adespoton@lemmy.ca 3 weeks ago
They walk down runways and pose for magazines. Do they reason? Sometimes.
random_character_a@lemmy.world 3 weeks ago
Yeah. That’s because peoples unfortunate reality is a “model”.
theunknownmuncher@lemmy.world 3 weeks ago
It’s true that LLMs aren’t “aware” of what internal steps they are taking, so asking an LLM how they reasoned out an answer will just output text that statistically sounds right based on its training set, but to say something like “they can never reason” is provably false.
Its obvious that you have a bias and desperately want reality to confirm it, but there’s been significant research and progress in tracing internals of LLMs, that show logic, planning, and reasoning. Neural networks and very powerful, after all, you are one too. Can you reason?
glizzyguzzler@lemmy.blahaj.zone 3 weeks ago
Too deep on the AI propaganda there, it’s completing the next word. You can give the LLM base umpteen layers to make complicated connections, still ain’t thinking.
The LLM corpos trying to get nuclear plants to power their gigantic data centers while AAA devs aren’t trying to buy nuclear plants says that’s a straw man and you simultaneously also are wrong.
Using a pre-trained and memory-crushed LLM that can run on a small device won’t take up too much power. But that’s not what you’re thinking of. You’re thinking of the LLM only accessible via ChatGPT’s api that has a yuge context length and massive matrices that needs hilariously large amounts of RAM and compute power to execute. And it’s still a facsimile of thought.
It’s okay they suck and have very niche actual use cases - maybe it’ll get us to something better. But they ain’t gold, they ain’t smart, and they ain’t worth destroying the planet.
ohwhatfollyisman@lemmy.world 3 weeks ago
would there be a source for such research?
A_Union_of_Kobolds@lemmy.world 3 weeks ago
AnneBonny@lemmy.dbzer0.com 3 weeks ago
How would you prove that someone or something is capable of reasoning or thinking?
glizzyguzzler@lemmy.blahaj.zone 3 weeks ago
You can prove it’s not by doing some matrix multiplication and seeing its matrix multiplication. Much easier way to go about it
Treczoks@lemmy.world 3 weeks ago
I’ve read that article. They used something they called an “MRI for AIs”, and checked e.g. how an AI handled math questions, and then asked the AI how it came to that answer, and the pathways actually differed. While the AI talked about using a textbook answer, it actually did a different approach. That’s what I remember of that article.
But yes, it exists, and it is science, not TicTok
lgsp@feddit.it 3 weeks ago
Thank you. I found the article, linkin the OP
WolfLink@sh.itjust.works 3 weeks ago
The environmental toll doesn’t have to be that bad. You can get decent results from single high-end gaming GPU.
glizzyguzzler@lemmy.blahaj.zone 2 weeks ago
You can, but the stuff that’s really useful (very competent code completion) needs gigantic context lengths that even rich peeps with $2k GPUs can’t do. And that’s ignoring the training power and hardware costs to get the models.
Techbros chasing VC funding are pushing LLMs to the physical limit of what humanity can provide power and hardware-wise. Way less hype and letting them come to market organically in 5/10 years would give the LLMs a lot more power efficiency at the current context and depth limits. But that ain’t this timeline, we just got VC money looking to buy nuclear plants and fascists trying to subdue the US for the techbro oligarchs womp womp
just_another_person@lemmy.world 3 weeks ago
It’s a developer option that isn’t generally available on consumer-facing products. It’s literally just a debug log that outputs the steps to arrive at a response, nothing more.
It’s not about novel ideation or reasoning (programmatic neural networks don’t do that), but just an output of statistical data that says “Step was 90% certain, Step 2 was 89% certain…etc”
AnneBonny@lemmy.dbzer0.com 3 weeks ago
Who has claimed that LLMs have the capacity to reason?
theparadox@lemmy.world 3 weeks ago
More than enough people who claim to know how it works think it might be “evolving” into a sentient being inside it’s little black box. Example from a conversation I gave up on… sh.itjust.works/comment/18759960
adespoton@lemmy.ca 3 weeks ago
The study being referenced explains in detail why they can’t. So I’d say it’s Anthropic who stated LLMs don’t have the capacity to reason, and that’s what we’re discussing.
The popular media tends to go on and on about conflating AI with AGI and synthetic reasoning.