There really are 2 kinds of AI content, one-shot AI slop, and something that someone has gone back-and-forth on a few times to edit and suggest changes. The later would be very hard to tell from human generated text.
Comment on Here’s how to spot AI writing, according to Wikipedia
Blackfeathr@lemmy.world 4 days ago
I’d gotten really good at discerning a chatGPT bot from a human account just from years of catching bots on Reddit.
There’s a lot of red flags and tells that would be very hard to completely eradicate. There will always be an uncanny valley.
Jason2357@lemmy.ca 4 days ago
Perspectivist@feddit.uk 4 days ago
You think you have - but there’s really no way of knowing.
Just because someone writes like a bot doesn’t mean they actually are one. Feeling like “you’ve caught one” doesn’t mean you did - it just means you think you did. You might have been wrong, but you never got confirmation to know for sure, so you have no real basis for judging how good your detection rate actually is. It’s effectively begging the question - using your original assumption as “proof” without actual verification.
And then there’s the classic toupee fallacy: “All toupees look fake - I’ve never seen one that didn’t.” That just means you’re good at spotting bad toupees. You can’t generalize from that and claim you’re good at detecting toupees in general, because all the good ones slip right past you unnoticed.
AwesomeLowlander@sh.itjust.works 4 days ago
Studies have shown we’re pretty bad at detecting good AI stuff, regardless of how skilled we think we are. It’s the crappy AI slop that makes everybody think they’re Sherlock Holmes.
pineapple_pizza@lemmy.dexlit.xyz 4 days ago
Just to add to this. I’ve noticed some of my co workers who have developing English skills sound like LLMs sometimes. I imagine this is probably because they’ve only wrote English in a school or work setting and never for personal communication.
…or maybe they’re all just using LLMs idk
Blackfeathr@lemmy.world 4 days ago
I mean I wasn’t going around accusing everyone of being a bot or thinking that I was right all the time. I did have a few false positives and owned up to it. But once you see the pattern of behavior (big gap on joined date vs first active date, only posting in karma farming subs or subs known to have high bot populations) and accounts literally keeping the “as a large language model…” Or “Okay, here’s a supportive Reddit-style comment with some minor spelling mistakes…” In some of their comments.
I have screenshots to prove it but if you really believe I don’t know what I’m talking about then there’s really nothing I can say to dissuade that.
null@lemmy.nullspace.lol 4 days ago
I don’t think anyone is questioning your ability to follow a hunch and get evidence to prove it.
The point is about how often you’re correct on your first guess.