Both sentences are true. And based on vocabulary of both, the model can output the following sentences:
- Cats have feathers.
- Birds have fur
This is not how the models are trained or work.
Both are false but the model doesn’t “know” it. All that it knows is that “have” is allowed to go after both “cats” and “birds”, and that both “feathers” and “fur” are allowed to go after “have”.
Demonstrably false. This isn’t how LLMs are trained or built.
Just considering the contextual relationships between word embeddings that are created during training is evidence enough. Those relationships from the multi-vector fields are an emergent property that doesn’t exist in the dataset.
If you want a better understanding of what I just said, take a look at this Computerphile video from four years ago. And this came out before the LLM hype and before ChatGPT 3, which was the big leap in LLMs.
KevonLooney@lemm.ee 2 months ago
It’s not just a predictive text program. That’s been around for decades. That’s a common misconception.
As I understand it, it uses statistics from the whole text to create new text. It would be very rare to output “cats have feathers” because that phrase doesn’t ever appear in the training data. Both words “have feathers” never follow “cats”.
skulblaka@sh.itjust.works 2 months ago
But the fact remains that it doesn’t know what a cat or a feather is. All of this is still based purely on statistical frequency and not at all on actual meanings.
vrighter@discuss.tchncs.de 2 months ago
and that is exactly how a predictive text algorithm works.
some tokens go in
they are processed by a deterministic, static statistical model, and a set of probabilities (always the same, deterministic, remember?) comes out.
pick the word with the highest probability, add it to your initial string and start over.
if you want variety, add some randomness and don’t just always pick the most probable next token.
Coincidentally, this is exactly how llms work. It’s a big markov chain, but with a novel lossy compression algorithm on its state transition table.
CeeBee_Eh@lemmy.world 2 months ago
Everyone who says this doesn’t actually understand how LLMs work.
Multivector word embeddings create emergent relationships that’s new knowledge that doesn’t exist in the training dataset.
Computerphile did a good video on this well before the LLM craze.
vrighter@discuss.tchncs.de 2 months ago
1 - a markov chain only takes previous tokens as input.
2 - It uses a function (in the mathematical sense, so same input results in same output, completely stateless) to generate a set of probabilities for what the next token might be.
3 - The most probable token is picked, else randomness (temperature) is inserted here to choose a different token occasionally.
an llm’s internals, the part that’s trained is literally the function used in step 2. You could have this function implemented a number of ways, ex you could buil a huge table and consult it. Or you could generate it somehow. You could train a big neural network that takes previous tokens as input, and outputs probabilities of tokens as output. It can be very smart and notice correlations, but ultimately it generates a (virtual) huge static table. This is a completely deterministic process. A trained NN is still a (huge) mathematical function. So the big network that they spend resources training is basically the function used in step 2.
Step 3 is the cause of hallucinations. It’s the only nondeterministic part.l. No matter how smarter the neural network gets, the hallucinations are introduced mainly in step 3. So no, they won’t be solving the LLM hallucination problem anytime soon.
barsoap@lemm.ee 2 months ago
Eh but LLMs abstract. It has seen “<animal> have feathers” and “<animal> have fur” quite a lot of times. The problem isn’t that LLMs can’t reason at all, the problem is that they do employ techniques used in proper reasoning, in particular tracking context throughout the text (cross-attention) but lack techniques necessary for the whole thing, instead relying on confabulation to sound convincing regardless of the BS they spout. Suffices to emulate an Etonian, but that’s not a high standard.
FaceDeer@fedia.io 2 months ago
Workarounds for those sorts of limitations have been developed, though. Chain-of-thought prompting has been around for a while now, and I recall recently seeing an article about a model that had that built right into it; it had been trained to use <thought></thought> tags to enclose invisible chunks of its output that would be hidden from the end user but would be used by the AI to work its way through a problem. So if you asked it whether cats had feathers it might respond "<thought>Feathers only grow on birds and dinosaurs. Cats are mammals.</thought> No, cats don't have feathers." And you'd only see the latter bit. It was a pretty neat approach to improving LLM reasoning.
WalnutLum@lemmy.ml 2 months ago
This isn’t really accurate either. At the moment of generation, an LLM only has context for the input string and the network of text tokens it’s been assigned. It pulls from a “pool” of these tokens based on what it’s already output and the input context, nothing more.
Most LLMs have what are called “Top P”, “Top K” etc, these are the number of tokens that it needs up selecting from based on the previous token, alongside the input tokens. It then randomly chooses one based on temperature settings.
It’s why if you turn these models’ temperature settings really high they output pure nonsense both conceptually and grammatically, because the tenuous thread linking the previous token’s context to the next token has been widened enough that it completely loses any semblance of cohesiveness.
lvxferre@mander.xyz 2 months ago
Your “ackshyually” is missing the point.