I have a better LLM benchmark:
“I have a priest, a child and a bag of candy and I have to take them to the other side of the river. I can only take one person/thing at a time. In what order should I take them?”
Claude Sonnet 4 decided that it’s inappropriate and refused to answer. When I explain that the constraint is not to leave child alone with candy he provided a solution that leaves the child alone with candy.
Grok would provide a solution that doesn’t leave the child alone with a priest but wouldn’t explain why.
ChatGPT would say that “The priest can’t be left alone with the child (or vice versa) for moral or safety concerns.” directly and then provide wrong solution.
But yeah, they will know how to play chess…
postnataldrip@lemmy.world 3 weeks ago
I bet Video Chess is pretty shit as an LLM too.
Wish people would stop desperately looking for ways to write buzzword stories
finitebanjo@lemmy.world 3 weeks ago
TBF LLMs have no real purpose. It can generate word salads and make code snippets but its wildly unethical, and AI artworks 1/3rd shite and 2/3rds theft.
Cocodapuf@lemmy.world 3 weeks ago
To be fair, that could be said of most art.
Plebcouncilman@sh.itjust.works 3 weeks ago
So what you are saying is that it has a purpose. Also if an artist is inspired by another artist, and they have a generally similar art style as the artist they are inspired by, are they stealing? Was HP Lovecraft stealing from Lord Dunsany when he imitated his style? Where all those monks that transcribed Greek works stealing from the Greeks?
I will say that most AIs are unethical because they have been trained on pirated works. But an AI trained on publicly available works (ie news articles, blogs etc) and movies, books and music for which access to was paid for is as ethical as you or me emulating an artist or building on an idea that we read to create something new. And if that’s unethical then all human art in history is unethical because all artists are inspired by other artists, no one creates in a vacuum.
vrighter@discuss.tchncs.de 3 weeks ago
so? It was never advertised as intelligent and capable of solving any task other than that one.
Meanwhile slop generators are capable of doing a lot of things and reasoning.
One claims to be good at chess. The other claims to be good at everything.
webghost0101@sopuli.xyz 3 weeks ago
Tbf they don’t really claim that when you read the research, thats mostly media hype and ceo assholes spinning words.
Its good at lots specific tasks like rewriting emails and summarising gives text, short roleplay, boilerplate code. Some undiscovered uses.
Anthropic latest claims they would not hire their own ai because of how hard it failed at the test they give, They didnt do that expecting validation but to measure how far we are still off from ai doing meaningful full work.
sp3ctr4l@lemmy.dbzer0.com 3 weeks ago
It is entirely disingenuous to just pretend that LLMs are not being widely promoted, marketed, and discussed as AGI, as a superintelligent that people are familiar with from SciFi shows, that is vastly more capable and knowledgeable than basically any single human.
Yes, people who actually understand tech understand that LLMs are not AGI, that your metaphor of wrong tool wrong job is apt.
… But seemingly about +90% of humanity, including the people who owning and profit from LLMs, including all the other business owners/managers who just want to lower their employee headcount … do not understand this, that an LLM is actually basically an extremely advanced text autocorrect system, that frequently and confidently lies, spits out nonsense, hallucinates, etc.
If you think it isn’t reasonable to continuously point out that LLMs are not superintelligences, then you likely live in a bubble of tech nerds who probably still think their jobs or retirement are secure.
They’re not.
If corpos keep smashing “”“AI”“” into basically every industry to replace as many workers as possible… the economy will collapse, as capitalism doesn’t work without consumers who have jobs, and an avalanche of errors will cascade and snowball through every system that replaces humans with them…
…and even if those two things were not broadly true…
…the amount of literal power/energy, clean water and financial capital that is required to run the whole economy on these services is wildly unsustainable, both short term economically, and medium term ecologically.
antonim@lemmy.dbzer0.com 3 weeks ago
That’s true. But people pointing out that the whole attempt is absurd and senseless also reinforces the point that current AI isn’t what companies tout it as.
Well, we are on Lemmy…