Yeah. This is a way bigger problem with this article than anything else. The entier thing hinges on their AI-detecting AI working. I have looked into how effective these kinds of tools are because it has come up at my work, and independent review of them suggests they’re, like, 3-5 times worse than the (already pretty bad) accuracy rates they claim, and disproportionatly flag non-native English speakers as AI generated. So, I’m highly skeptical of this claim as well.
Comment on Study of 8k Posts Suggests 40+% of Facebook Posts are AI-Generated
morrowind@lemmy.ml 2 days ago
Keep in mind this is for AI generated TEXT, not the images everyone is talking about in this thread.
Also they used an automated tool, all of which have very high error rates, because detecting AI text is a fundamentally impossible task
PumpkinSkink@lemmy.world 1 day ago
addie@feddit.uk 2 days ago
AI does give itself away over “longer” posts, and if the tool makes about an equal number of false positives to false negatives then it should even itself out in the long run. (I’d have liked more than 9K “tests” for it to average out, but even so.) If they had the edit history for the post, which they didn’t, then it’s more obvious. AI will either copy-paste the whole thing in in one go, or will generate a word at a time at a fairly constant rate. Humans will stop and think, go back and edit things, all of that.
I was asked to do some job interviews recently; the tech test had such an “animated playback”, and the difference between a human doing it legitimately and someone using AI to copy-paste the answer was surprisingly obvious. The tech test questions were nothing to do with the job role at hand and were causing us to select for the wrong candidates completely, but that’s more a problem with our HR being blindly in love with AI and “technical solutions to human problems”.
“Absolute certainty” is impossible, but balance of probabilities will do if you’re just wanting an estimate like they have here.
morrowind@lemmy.ml 2 days ago
I have no idea whether the probabilities are balanced. They claim 5% was AI even before chatgpt was released, which seems pretty off. No one was using LLMs before chatgpt went viral except for researchers.
GenosseFlosse@feddit.org 2 days ago
Chatbots doesn’t mean that they have a real conversation. Some just spammed links from a list of canned responses, or just upvoted the other chat bots to get more visibility, or the just reposted a comment from another user.
szczuroarturo@programming.dev 2 days ago
Im pretty sure chatbots were a thing before AI. They certainly werent as smart but they did exists.
ubergeek@lemmy.today 1 day ago
chat bots have been a thing, for a long time. I mean, a half decently trained Markov can handle social media postings and replies