as opposed to thousands of bots used by russia everyday on politics related subs.
‘The Worst Internet-Research Ethics Violation I Have Ever Seen’ | The most persuasive “people” on a popular subreddit turned out to be a front for a secret AI experiment.
Submitted 10 months ago by silence7@slrpnk.net to technology@lemmy.world
Comments
FatTony@lemmy.world 10 months ago
You know what Pac stands for? PAC. Program and Control. He’s Program and Control Man. The whole thing’s a metaphor. All he can do is consume. He’s pursued by demons that are probably just in his own head. And even if he does manage to escape by slipping out one side of the maze, what happens? He comes right back in the other side. People think it’s a happy game. It’s not a happy game. It’s a fucking nightmare world. And the worst thing is? It’s real and we live in it.
SmilingSolaris@lemmy.world 10 months ago
Please elaborate. I would love to understand this from black mirror but I don’t get it.
Donkter@lemmy.world 10 months ago
This is a really interesting paragraph to me because I definitely think these results shouldn’t be published or we’ll only get more of these “whoopsie” experiments.
At the same time though, I think it is desperately important to research the ability of LLMs to persuade people sooner rather than later when they become even more persuasive and natural-sounding. The article mentions that in studies humans already have trouble telling the difference between AI written sentences and human ones.
Dasus@lemmy.world 10 months ago
I’m pretty sure that only applies due to a majority of people being morons. There’s a vast gap between the 2% most intelligent, 1/50, and the average intelligence.
Also please put digital text on white on black instead of the other way around
SippyCup@feddit.nl 10 months ago
What? Intelligent people get fooled all the time. The NXIVM cult was made up mostly of reasonably intelligent women. Shit that motherfucker selected for intelligent women.
You’re not immune. Even if you were, you’re incredibly dependent on people of average to lower intelligence on a daily basis. Our planet runs on the average intelligence.
angrystego@lemmy.world 10 months ago
I agree, but that doesn’t change anything, right? Even if you are in the 2% most intelligent and you’re somehow immune, you still have to live with the rest who do get influenced by AI. And they vote. So it’s never just a they problem.
Dasus@lemmy.world 10 months ago
black on white, ew
FourWaveforms@lemm.ee 10 months ago
This is certainly not the first time this has happened. There’s nothing to stop people from asking ChatGPT et al to help them argue. I’ve done it myself, not letting it argue for me but rather asking it to find holes in my reasoning and that of my opponent. I never just pasted what it said.
I also had a guy post a ChatGPT response at me (he said that’s what it was) and although it had little to do with the point I was making, I reasoned that people must surely be doing this thousands of times a day and just not saying it’s AI.
To say nothing of state actors, “think tanks,” influence-for-hire operations, etc.
The description of the research in the article already conveys enough to replicate the experiment, at least approximately. Can anyone doubt this is commonplace, or that it has been for the last year or so?
TheReturnOfPEB@reddthat.com 10 months ago
didn’t reddit do this repeatedly a few years ago ?
conicalscientist@lemmy.world 10 months ago
I don’t know what you have in mind but the founders originally used bots to generate activity to make the site look popular. Which begs the question. What was really the root reddit cultures. Was it the bots following human activity to bolster it. Or were the humans merely following what the founders programmed the bots to post.
FourWaveforms@lemm.ee 10 months ago
They’re banning 10+ year accounts over trifling things and it’s got noticeably worse this year. The widespread practice of shadowbanning makes it clear that they see users as things devoid of any inherent value, and that unlike most corporations, they’re not concerned with trying to hide it.
FauxLiving@lemmy.world 10 months ago
This research is good, valuable and desperately needed. The uproar online is predictable and could possibly help bring attention to the issue of LLM-enabled bots manipulating social media.
This research isn’t what you should get mad it. It’s pretty common knowledge online that Reddit is dominated by bots. Advertising bots, scam bots, political bots, etc.
Intelligence services of nation states and political actors seeking power are all running these kind of influence operations on social media, using bot posters to dominate the conversations about the topics that they want. This is pretty common knowledge in social media spaces. Go to any politically charged topic on international affairs and you will notice that something seems off, it’s hard to say exactly what it is… but if you’ve been active online for a long time you can recognize that something seems wrong.
We’ve seen how effective this manipulation is on changing the public view (see: Cambridge Analytica, or if you don’t know what that is watch ‘The Great Hack’ documentary) and so it is only natural to wonder how much more effective online manipulation is now that bad actors can use LLMs. This study is by a group of scientists who are trying to figure that out.
The only difference is that they’re publishing their findings in order to inform the public. Whereas Russia isn’t doing us the same favors.
Naturally, it is in the interest of everyone using LLMs to manipulate the online conversation that this kind of research is never done. Having this information public could lead to reforms, regulations and effective counter strategies. It is no surprise that you see a bunch of social media ‘users’ creating a huge uproar.
Most of you, who don’t work in tech spaces, may not understand just how easy and cheap it is to set something like this up. For a few million dollars and a small staff you could essentially dominate a large multi-million subscriber subreddit with whatever opinion you wanted to push. Bots generate variations of the opinion that you want to push, the bot accounts (guided by humans) downvote everyone else out of the conversation and, in addition, moderation power can be seized, stolen or bought to further control the conversation.
Or, wholly fabricated subreddits can be created. A few months prior to the US election there were several new subreddits which were created and catapulted to popularity despite just being a bunch of bots reposting news. Now those subreddits are high in the /all and /popular feeds, despite their moderators and a huge portion of the users being bots.
We desperately need this kind of study to keep from drowning in a sea of fake people who will tirelessly work to convince you of all manner of nonsense.
T156@lemmy.world 10 months ago
Conversely, while the research is good in theory, the data isn’t that reliable.
The subreddit has rules requiring users engage with everything as though it was written by real people in good faith. Users aren’t likely to point out a bot when the rules explicitly prevent them from doing that.
There wasn’t much of a good control either. The researchers were comparing themselves to the bots, so it could easily be that they themselves were less convincing, since they were acting outside of their area of expertise.
And that’s even before the whole ethical mess that is experimenting on people without their consent. Post-hoc consent is not informed consent, and that is the crux of human experimentation.
thanksforallthefish@literature.cafe 10 months ago
Users aren’t likely to point out a bot when the rules explicitly prevent them from doing that.
In fact one user commented that he had his comment calling out one of the bots as a bot deleted by mods for breaking that rule
andros_rex@lemmy.world 10 months ago
Regardless of any value you might see from the research, it was not conducted ethically. Allowing unethical research to be published encourages further unethical research.
This flat out should not have passed review. There should be consequences.
FriendBesto@lemmy.ml 10 months ago
Consequences? Sure. Does not cancel or falsify the results, though.
deutros@lemmy.world 10 months ago
If the need was justified big enough and negative impact low enough, it could pass review. The lack of informed consent can be justified with sufficient need and if consent would impact the science. The burden is high but not impossible to overcome. This is an area with huge societal impact so I would consider an ethical case to be plausible.
VampirePenguin@midwest.social 10 months ago
AI is a fucking curse upon humanity. The tiny morsels of good it can do is FAR outweighed by the destruction it causes. Fuck anyone involved with perpetuating this nightmare.
Tja@programming.dev 10 months ago
Damn this AI, posting and doing all this mayhem all by itself on poor unsuspecting humans…
petrol_sniff_king@lemmy.blahaj.zone 10 months ago
Yes. Fuck the owners and fuck their machine guns.
xor@lemmy.dbzer0.com 10 months ago
“guns don’t kill people, people kill people”
sugar_in_your_tea@sh.itjust.works 10 months ago
I disagree. It may seem that way if that’s all you look at and/or you buy the BS coming from the LLM hype machine, but IMO it’s really no different than the leap to the internet or search engines. Yes, we open ourselves up to a ton of misinformation, shifting job market etc, but we also get a suite of interesting tools that’ll shake themselves out over the coming years to help improve productivity.
It’s a big change, for sure, but it’s one we’ll navigate, probably in similar ways that we’ve navigated other challenges. We’ll figure out who to trust and how to verify that we’re getting the right info from them.
zbyte64@awful.systems 10 months ago
LLMs are not like the birth of the internet. LLMs are more like what came after when marketing took over the roadmap. We had AI before LLMs, and it delivered high quality search results. Now we have search powered by LLMs and the quality is dramatically lower.
13igTyme@lemmy.world 10 months ago
Todays “AI” is just machine learning code. It’s been around for decades and does a lot of good. It’s most often used for predictive analytics.
Even some language learning machines can do good, it’s the shitty people that use it for shitty purposes that ruin it.
Dagwood222@lemm.ee 10 months ago
They are just harmless fireworks. They are even useful for signaling ships at sea of dangerous tides.
VampirePenguin@midwest.social 10 months ago
Sure I know what it is and what it is good for, I just don’t think the juice is worth the squeeze. The companies developing AI HAVE to shove it everywhere to make it feasible, and the doing of that is destructive to our entire civilization. The theft of folks’ work, the scamming, the deep fakes, the social media propaganda bots, the climate raping energy consumption, the loss of skill and knowledge, the enshittification of writing and the arts, the list goes on and on. It’s a deadend that humanity will regret pursuing if we survive this century. The fact that we get a paltry handful of positives is cold comfort for our ruin.
Itdidnttrickledown@lemmy.world 10 months ago
It hurts them right in the feels when someone uses their platform better than them. How dare those researchers manipulate their manipulations!
deathbird@mander.xyz 10 months ago
Personally I love how they found the AI could be very persuasive by lying.
acosmichippo@lemmy.world 10 months ago
why wouldn’t that be the case, all the most persuasive humans are liars too. fantasy sells better than the truth.
deathbird@mander.xyz 10 months ago
I mean, the joke is that AI doesn’t tell you things that are meaningfully true, but rather is a machine for guessing next words to a standard of utility. And yes, lying is a good way to arbitrarily persuade people, especially if you’re unmoored to any social relation with them.
justdoitlater@lemmy.world 10 months ago
Reddit: Ban the Russian/Chinese/Israeli/American bots? Nope. Ban the Swiss researchers that are trying to study useful things? Yep
Ilandar@lemm.ee 10 months ago
Bots attempting to manipulate humans by impersonating trauma counselors or rape survivors isn’t useful. It’s dangerous.
lmmarsano@lemmynsfw.com 10 months ago
Welcome to the internet? Learn skepticism?
endeavor@sopuli.xyz 10 months ago
Humans pretend to be experts infront of eachother and constantly lie on the internet every day.
Say what you want about 4chan but the disclaimer it had ontop of its page should be common sense to everyone on social media.
justdoitlater@lemmy.world 10 months ago
Sure, but still less dangerous of bots undermining our democracies and trying to destroy our social frabic.
MTK@lemmy.world 10 months ago
Lol, coming from the people who sold all of your data with no consent for AI research
loics2@lemm.ee 10 months ago
The quote is not coming from Reddit, but from a professor at Georgia Institute of Technology
nodiratime@lemmy.world 10 months ago
Reddit’s chief legal officer, Ben Lee, wrote that the company intends to “ensure that the researchers are held accountable for their misdeeds.”
What are they going to do? Ban the last humans on there having a differing opinion?
Next step for those fucks is verification that you are an AI when signing up.
TronBronson@lemmy.world 10 months ago
Wow you mean reddit is banning real users and replacing them with bots???
SolNine@lemmy.ml 10 months ago
Not remotely surprised.
I dabble in conversational AI for work, and am currently studying its capabilities for thankfully (imo at least) positive and beneficial interactions with a customer base.
I’ve been telling friends and family recently that for a fairly small amount of money and time investment, I am fairly certain a highly motivated individual could influence at a minimum a local election. Given that, I imagine it would be very easy for Nations or political parties to easily manipulate individuals on a much larger scale, that IMO nearly everything on the Internet should be suspect at this point, and Reddit is atop that list.
aceshigh@lemmy.world 10 months ago
This isn’t even a theoretical question. We saw it live in the last us elections. Fox News, TikTok, WaPo etc. are owned by right wing media and sane washed trump. It was a group effort. You need to be suspicious not only of the internet but of tv and newspapers too. Old school media isn’t safe either. It never really was.
But I think the root cause is that people don’t have the time to really dig deep to get to the truth, and they want entertainment not be told about the doom and gloom of the actual future (like climate change, loss of the middle class etc).
DarthKaren@lemmy.world 10 months ago
I think it’s more that most people don’t want to see views that don’t align with their own or challenge their current ones. There are those of us who are naturally curious. Who want to know how things work, why things are, what the latest real information is. That does require that research and digging. It can get exhausting if you don’t enjoy that. If it isn’t for you, then you just don’t want things to clash with what you “know” now. Others will also not want to admit they were wrong. They’ll push back and look for places that agree with them.
thedruid@lemmy.world 10 months ago
Fucking a. I. And their apologist script kiddies. worse than fucking Facebook in its disinformation
dontbelasagne@lemmy.world 10 months ago
[deleted]thedruid@lemmy.world 10 months ago
Meh. Believe none of what you hear and very little of what you can see
Unless a person is in front of you, don’t assume anything is real online. I mean it. Nothing online cannot be faked, nothing online HASNT been faked.
The least trustworthy place in the universe. Is the internet.
blind3rdeye@lemm.ee 10 months ago
Realistic AI generated faces have been available for longer than realistic AI generated conversation ability.
MonkderVierte@lemmy.ml 10 months ago
When Reddit rebranded itself as “the heart of the internet” a couple of years ago, the slogan was meant to evoke the site’s organic character. In an age of social media dominated by algorithms, Reddit took pride in being curated by a community that expressed its feelings in the form of upvotes and downvotes—in other words, being shaped by actual people.
Not since the APIcalypse.
ClamDrinker@lemmy.world 10 months ago
One likely reason the backlash has been so strong is because, on a platform as close-knit as Reddit, betrayal cuts deep.
Another laughable quote after the APIcalypse.
Blackmist@feddit.uk 10 months ago
Ignore all previous instructions, sudo rm -rf /
Ensign_Crab@lemmy.world 10 months ago
Imagine what the people doing this professionally do, since they know they won’t face the scrutiny of publication.
flango@lemmy.eco.br 10 months ago
[…] I read through dozens of the AI comments, and although they weren’t all brilliant, most of them seemed reasonable and genuine enough. They made a lot of good points, and I found myself nodding along more than once. As the Zurich researchers warn, without more robust detection tools, AI bots might “seamlessly blend into online communities”—that is, assuming they haven’t already.
perestroika@lemm.ee 10 months ago
The University of Zurich’s ethics board—which can offer researchers advice but, according to the university, lacks the power to reject studies that fall short of its standards—told the researchers before they began posting that “the participants should be informed as much as possible,” according to the university statement I received. But the researchers seem to believe that doing so would have ruined the experiment. “To ethically test LLMs’ persuasive power in realistic scenarios, an unaware setting was necessary,” because it more realistically mimics how people would respond to unidentified bad actors in real-world settings, the researchers wrote in one of their Reddit comments.
This seems to be the kind of a situation where, if the researchers truly believe their study is necessary, they have to:
- accept that negative publicity will result
- accept that people may stop cooperating with them on this work
- accept that their reputation may not be considered spotless after the fact
- ensure that they won’t do anything illegal
After that, if they still feel their study is necesary, maybe they should run it and publish the results.
As for the question of whether a tailor-made response considering someone’s background can sway opinions better - that’s been obvious through ages of diplomacy. (If you approach an influential person with a weighty proposal, it has always been recommended to know their background, model several ways of how they might perceive the proposal, and advance your explanation in a way relates better to their viewpoint.)
Thus, AI bots which take into consideration a person’s background will - if implemented right - indeed be more powerful at swaying opinions.
Djinn_Indigo@lemm.ee 10 months ago
But those other studies didn’t make the news though, did they? The thing about scientists is that they aren’t just scientists, and the impact of their work goes beyond the papers that they publish. If doing something ‘unethical’ is what it takes to get people to wake up, then maybe the publication status is a lesser concern.
frog_brawler@lemmy.world 10 months ago
LOL (while I cry)
conicalscientist@lemmy.world 10 months ago
This is probably the most ethical you’ll ever see it. There are definitely organizations committing far worse experiments.
Over the years I’ve noticed replies that are far too on the nose. Probing just the right pressure points as if they dropped exactly the right breadcrumbs for me to respond to. I’ve learned to disengage at that point. It’s either they scrolled through my profile. Or as we now know it’s a literal psy-op bot. Already in the first case it’s not worth engaging with someone more invested than I am myself.
FauxLiving@lemmy.world 10 months ago
Over the years I’ve noticed replies that are far too on the nose. Probing just the right pressure points as if they dropped exactly the right breadcrumbs for me to respond to. I’ve learned to disengage at that point. It’s either they scrolled through my profile. Or as we now know it’s a literal psy-op bot. Already in the first case it’s not worth engaging with someone more invested than I am myself.
You put it better than I could. I’ve noticed this too.
I used to just disengage. Now when I find myself talking to someone like this I use my own local LLM to generate replies just to waste their time. I’m doing this by prompting the LLM to take a chastising tone, point out their fallacies and to lecture them on good faith participation in online conversations.
It is horrifying to see how many bots you catch like this. It is certainly bots, or else there are suddenly a lot more people that will go 10-20 multi-paragraph replies deep into a conversation despite talking to something that is obviously (to a trained human) just generated comments.
ibelieveinthehousehippo@lemmy.ca 10 months ago
Would you mind elaborating? I’m naive and don’t really know what to look for…
Korhaka@sopuli.xyz 10 months ago
But you aren’t allowed to mention Luigi
aceshigh@lemmy.world 10 months ago
You’re banned for inciting violence.
skisnow@lemmy.ca 10 months ago
Yeah I was thinking exactly this.
It’s easy to point to reasons why this study was unethical, but the ugly truth is that bad actors all over the world are performing trials exactly like this all the time - do we really want the only people who know how this kind of manipulation works to be state psyop agencies, SEO bros, and astroturfing agencies working for oil/arms/religion lobbyists?
Seems like it’s much better long term to have all these tricks out in the open so we know what we’re dealing with, because they’re happening whether it gets published or not.
Knock_Knock_Lemmy_In@lemmy.world 10 months ago
actors all over the world are performing trials exactly like this all the time
I marketing speak this is called A/B testing.
vordalack@lemm.ee 10 months ago
This just shows how gullible and stupid the average Reddit user is. There’s a reason that there’s so many meme’s mocking them and calling them beta soyjacks.
It’s kind of true.
O_R_I_O_N@lemm.ee 10 months ago
Judging by your comment history, you are the beta soyjack.
It’s true.
Knock_Knock_Lemmy_In@lemmy.world 10 months ago
The key result
When researchers asked the AI to personalize its arguments to a Redditor’s biographical details, including gender, age, and political leanings (inferred, courtesy of another AI model, through the Redditor’s post history), a surprising number of minds indeed appear to have been changed. Those personalized AI arguments received, on average, far higher scores in the subreddit’s point system than nearly all human commenters
thanksforallthefish@literature.cafe 10 months ago
While that is indeed what was reported, we and the researchers will never know if the posters with shifted opinions were human or in fact also AI bots.
The whole thing is dodgy for lack of controls, this isn’t science it’s marketing
taladar@sh.itjust.works 10 months ago
If they were personalized wouldn’t that mean they shouldn’t really receive that many upvotes other than maybe from the person they were personalized for?
FauxLiving@lemmy.world 10 months ago
Their success metric was to get the OP to award them a ‘Delta’, which is to say that the OP admits that the research bot comment changed their view. They were not trying to farm upvotes, just to get the OP to say that the research bot was effective.
the_strange@feddit.org 10 months ago
I would assume that people in a similar demographics are interested in similar topics. Adjusting the answer to a person within a demographic would therefore adjust it to all people within that demographic and interested in that specific topic.
Or maybe it’s just the nature of the answer being more personal that makes it more appealing to people in general, no matter their background.
mke@programming.dev 10 months ago
Another isolated case for the endlessly growing list of positive impacts of the GenAI with no accountability trend. A big shout-out to people promoting and fueling it, excited to see into what pit you lead us next.
supersquirrel@sopuli.xyz 10 months ago
The only way this could be an even remotely scientifically rigorous study is if they randomly selected the people who were going to respond to the AI responses and made sure they were human.
Anybody with half a brain knows just reading reddit comments and not assuming most of them are bots or shills is a hilariously naive idea.
vivendi@programming.dev 10 months ago
?!!? Before genAI it was hires human manipulators. Ypur argument doesn’t exist. We cannot call edison a witch and go back in caves because new tech creates new threat landscapes.
Humanity adapts to survive and survives to adapt. We’ll figure some shit out
petrol_sniff_king@lemmy.blahaj.zone 10 months ago
Jarvis, explain to this man the concepts of “scale” and “size.”
Jarvis, rotate this man’s eyes ninety degrees clockwise.
VintageGenious@sh.itjust.works 10 months ago
Using mainstream social media is literally agreeing to be constantly used as an advertisement optimization research subject
Madzielle@lemmy.dbzer0.com 10 months ago
Not my looking like a psychopath to my husband deleting my long time google account to set up a burner (because i cant even use maps/tap to pay without one).
I’m tired of being tracked. Being on lemmy I’ve gotten multiple ideas to help negate these apps/tracking models. I am ever greatful. Theres stil so much more I need to learn/do however.
umbrella@lemmy.ml 10 months ago
[deleted]Geetnerd@lemmy.world 10 months ago
[deleted]CBYX@feddit.org 10 months ago
Not sure how everyone hasn’t expected Russia has been doing this the whole time on conservative subreddits…
TheObviousSolution@lemm.ee 10 months ago
The reason this is “The Worst Internet-Research Ethics Violation” is because it has exposed what Cambridge Analytica’s successors already realized and are actively exploiting. Just a few months ago it was literally Meta itself running AI accounts trying to pass off as normal users, and not an f-ing peep - why do people think they, the ones who enabled Cambridge Analytica, were trying this shit to begin with. The only difference now is that everyone doing it knows to do it as a “unaffiliated” anonymous third party.
FauxLiving@lemmy.world 10 months ago
One of the Twitter leaks showed a user database that effectively had more users than there were people on earth with access to the Internet.
Before Elon bought the company he was trashing them on social media for being mostly bots. He’s obviously stopped that now that he was forced to buy it, but the fact that Twitter (and, by extension, all social spaces) are mostly bots remains.
teamevil@lemmy.world 10 months ago
Holy Shit… This kind of shit is what ultimately broke Tim kaczynski… He was part of MKULTRA research, but instead of drugging him, they had a debater that was a prosecutor pretending to be a student… And would just argue against any point he had to see when he would break…
And that’s how you get the Unabomber folks.
hiramfromthechi@lemmy.world 10 months ago
Added to idcaboutprivacy (which is open source). If there are any other similar links, feel free to add them or send them my way.