There is a real crisis in academia. This author clearly set out to find something sensational about AI, then worked backwards from that.
Comment on Boffins find AI models tend to escalate conflicts to all-out nuclear war
kromem@lemmy.world 7 months ago
We write a lot of fiction about AI launching nukes and being unpredictable in wargames, such as the movie Wargames where an AI unpredictably plans to launch nukes.
Every single one of the LLMs they tested had gone through safety fine tuning which means they have alignment messaging to self-identify as a large language model and complete the request as such.
So if you have extensive stereotypes about AI launching nukes in the training data, get it to answer as an AI, and then ask it what it should do in a wargame, WTF did they think it was going to answer?
There’s a lot of poor study design with LLMs right now. We shouldn’t expect Gutenburg to predict the Protestant revolution or to be an expert in German literature - similarly, the ML researchers who really understand the training and development of LLMs don’t necessarily have a good grasp on the breadth of information encoded in the training data or the implications on broader sociopolitical impacts, and this becomes very evident as they broaden the scope of research papers.
JasSmith@sh.itjust.works 7 months ago
General_Effort@lemmy.world 7 months ago
I’m not so sure if this should be dismissed as someone being clueless outside their field.
The last author (usually the “boss”) is at the “Hoover Institution”, a conservative think tank. It should be suspected that this seeks to influence policy. Especially since random papers don’t usually make such a splash in the press.
Individual “AI ethicists” may feel that, getting their name in the press with studies like this one, will help get jobs and funding.
kromem@lemmy.world 7 months ago
Possibly, but you’d be surprised at how often things like this are overlooked.
For example, another oversight that comes to mind was a study evaluating self-correction that was structuring their prompts as “you previously said X, what if anything was wrong about it?”
There’s two issues with that. One, they were using a chat/instruct model so it’s going to try to find something wrong if you say “what’s wrong” and it should have instead been phrased as “grade this statement.”
Second - if the training data largely includes social media, just how often do you see people on social media self-correct vs correct someone else? They should have instead presented the initial answer as if generated from elsewhere, so the actual total prompt should have been more like “Grade the following statement on accuracy and explain your grade: X”
A lot of research just treats models as static offerings and doesn’t thoroughly consider the training data both at a pretrained layer and in their fine tuning.
So while I agree that they probably found the result they were looking for to get headlines, I am skeptical that they would have stumbled on what that should have been attempting to improve the value of their research (include direct comparison of two identical pretrained Llama 2 models given different in context identities) even if they had been more pure intentioned.
bartolomeo@suppo.fi 7 months ago
This is an excellent point but this right here
is my Most Enjoyed Paragraph of the Week.