Submitted 1 year ago by popcar2@programming.dev to technology@lemmy.world
Previous posts: programming.dev/post/3974121 and programming.dev/post/3974080
Thanks for all the answers, here are the results for the survey in case you were wondering how you did!
I still don’t believe the avocado comic is one-shot AI-generated. Composited from multiple outputs, sure. But I have not once seen generative AI produce an image that includes properly rendered text like this.
Bing image creator uses the new DALL-E model which does hands and text pretty good.
People forget just how fast this tech is evolving
Image generation tech has gone crazy over the past year and a half or so. At the speed it’s improving I wouldn’t rule out the possibility.
Here’s a paper from this year discussing text generation within images (it’s very possible these methods aren’t SOTA anymore – that’s how fast this field is moving): …thecvf.com/…/Rodriguez_OCR-VQGAN_Taming_Text-Wit…
Yeah I’m sceptical too, what tool and prompt was used to produce this?
Its Dalle 3 its not that difficult to generate something like that using dalle 3 here’s some shreks I generated as a showcase Shrek 1 inage
All of these are just generated nothing else
I found the avocado comic the easiest to tell, since the missing eyebrow was so insanely out of place.
Its not that difficult to generate something like that using dalle 3 here’s some shreks I generated as a showcase Shrek 1 inage
All of these are just generated nothing else
Prompt and tool links? I know there are tools that try to pick out label text in the prompt and composite it after the fact, but I don’t consider this one-shot AI generated, even if it’s a single tool from the user’s perspective.
Did you not check for a correlation between profession and accuracy of guesses?
I have. Disappointingly there isn’t much difference, the people working in CS have a 9.59 avg while the people that aren’t have a 9.61 avg.
There is a difference in people that have used AI gen before. People that have got a 9.70 avg, while people that haven’t have a 9.39 avg score. I’ll update the post to add this.
Can we get the raw data set? / could you make it open? I have academic use for it.
Sampling from Lemmy is going to severely skew the respondent population towards more technical people, even if their official profession is not technical.
If you do another one of these, I would like to see artist vs non-artist. If anything I feel like they would have the most experience with regular art, and thus most able to spot incongruency in AI art.
Something I’d be interested in is restricting the “Are you in computer science?” question to AI related fields, rather than the whole of CS, which is about as broad a field as social science, of which neural networks are a tiny sliver of a tiny sliver
Especially depending on the nation or district a person lives in, where CS can have even broader implications like everything from IT Support to Engineering.
And this is why AI detector software is probably impossible.
Just about everything we make computers do is something we’re also capable of; slower, yes, and probably less accurately or with some other downside, but we can do it. We at least know how. We can’t program software or train neutral networks to do something that we have no idea how to do.
If this problem is ever solved, it’s probably going to require a whole new form of software engineering.
And this is why AI detector software is probably impossible.
What exactly is “this”?
Just about everything we make computers do is something we’re also capable of; slower, yes, and probably less accurately or with some other downside, but we can do it. We at least know how.
There are things computers can do better than humans, like memorizing, or precision (also both combined). For all the rest, while I agree in theory we could be on par, in practice it matters a lot that things happen in reality. There often is only a finite window to analyze and react and if you’re slower, it’s as good as if you knew nothing. Being good / being able to do something often means doing it in time.
We can’t program software or train neutral networks to do something that we have no idea how to do.
Machine learning does that. We don’t know how all these layers and neurons work, we could not build the network from scratch. We cannot engineer/build/create the correct weights, but we can approach them in training.
Also look at Generative Adversarial Networks (GANs). The adversarial part is literally to train a network to detect bad AI generated output, and tweak the generative part based on that error to produce better output, rinse and repeat. Note this by definition includes a (specific) AI detector software, it requires it to work.
What exactly is “this”?
The results of this survey showing that humans are no better than a coin flip.
while I agree in theory we could be on par, in practice it matters a lot that things happen in reality.
I didn’t say “on par.” I said we know how. I didn’t say we were capable, but we know how it would be done. With AI detection, we have no idea how it would be done.
Machine learning does that.
No it doesn’t. It speedruns the tedious parts of writing algorithms, but we still need to be able to compose the problem and tell the network what an acceptable solution would be.
Also look at Generative Adversarial Networks (GANs). […] this by definition includes a (specific) AI detector software, it requires it to work.
Several startups, existing tech giants, AI companies, and university research departments have tried. There are literally millions on the line. All they’ve managed to do is get students incorrectly suspended from school, misidentify the US Constitution as AI output, and get a network really good at identifying training data and absolutely useless at identifying real world data.
Note that I said that this is probably impossible, only because we’ve never done it before and the experiments undertaken so far by some of the most brilliant people in the world have yielded useless results. I could be wrong. But the evidence so far seems to indicate otherwise.
I don’t know… My computer can do crazy math like 13+64 and other impossible calculations like that.
Wow, what a result. Slight right skew but almost normally distributed around the exact expected value for pure guessing.
I feel like the images selected were pretty vague. Like if you have a picture of a stick man and ask if a human or computer drew it. Some styles aew just hard to tell
You could count the fingers but then again my preschooler would have drawn anywhere from 4 to 40.
Imo, 3,17,18 were obviously AI imo (based on what I've seen from AI art generators in the past). But whatever original art those are based on, I'd probably also flag as obviously AI. The rest I was basically guessing at random. Especially the sketches.
I’d be curious about 18, what makes it obviously generated for you? Out of the ones not shown in the result, I got most right but not this one.
I was legitimately surprised by the man on a bench being human-made. His ankle is so thin! The woman in a bar/restaurant also surprised me because of her tiny finger.
Wow, I got a 12/20. I thought I would get less. I’m scared for the future of artists
Why? A lot of artists have adopted AI and use it as just another tool.
I’m sure artists can use it as another tool, but the problem comes when companies think they can get away with just using ai. Also, the ai has been trained using artwork without any artist permission
Which is an issue if those artists want to copyright their work. So far the US has maintained that AI generated art is not subject to copyright protection.
As with other AI-enhanced jobs, that probably still means less jobs in the long run.
Now one artist can make more art in the same time, or produce different styles which previously had required different artists.
Having used stable diffusion quite a bit, I suspect the data set here is using only the most difficult to distinguish photos. Most results are nowhere near as convincing as these. Notice the lack of hands. Still, this establishes that AI is capable of creating art that most people can’t tell apart from human made art, albeit with some trial and error and a lot of duds.
These images were fun, but we can’t draw any conclusions from it. They were clearly chosen to be hard to distinguish. It’s like picking 20 images of androgynous looking people and then asking everyone to identify them as women or men. The fact that success rate will be near 50% says nothing about the general skill of identifying gender.
I have it on very good authority from some very confident people that all ai art is garbage and easy to identify. So this is an excellent dataset to validate my priors.
Sketches are especially hard to tell apart because even humans put in extra lines and add embellishments here and there. I’m not surprised more than 70% of participants weren’t able to tell that one was generated.
Curious which man made image was most likely to be clarified as ai generated
and
About 20% got those correct as human-made.
My first impression was "AI" when I saw them, but I figured an AI would have put buildings on the road in the town and the 2nd one was weird but that parts fit together well enough.
I think it's because of the shadow on the ground being impossible
Thank you so much for sharing the results. Very interesting to see the outcome after participating in the survey.
Out of interest: do you know how many participants came from Lemmy compared to other platforms?
No idea, but I would assume most results are from here since Lemmy is where I got the most attention and feedback.
I’m found the survey through tumblr, i expect there’s quite a few from there.
I guess I should feel good with my 12/20 since it’s better than average.
What I learnt is that I’m bad at that task.
Oof. I got about 65% on the images I hadn’t seen in the post. I must be pretty close to being replaceable by an adversarial network.
I got 11/20 and there were a couple of guesses in there that I got right and wrong. Funny how there are some man-made ones that seem like AI. I think it’s the blurry/fuzziness maybe?
You put ‘tell’ twice in the title 😅
God DAMN it
I can’t tell if that would be more of a human or AI mistake 🧐
I did 12 and I thought it was a pretty bad result. Apparently I was wrong!
I tried to answer by instinct, without careful examination, to make it more realistic
15 here and when I reviewed the answers I realized most were sheer luck.
It’d be interesting to also see the two non-AI images the most people thought were.
Interesting. So you’ve given us a 50/ 50 chance. Usually you’ve given us the art that was used and then the AI has attempted it’s own version? Did you train the ai using the art ?
Are you allowed to do that ? Is the art in the public sphere?
I have no knowledge or understanding of ai.
No, the AI didn’t try to copy the other art that was included. I also don’t train the model myself, I just tell it to create an image similar to another one. For example the fourth picture I told it to create a rough sketch of a person sitting on a bench using an ink pen, then I went online and looked for a human-made one that’s of a similar style.
Ah ok. So it didn’t use those images to train on. Were those images hard to find ? Does it tell you what it uses to train ?
Could you credit the human artists somewhere?
I would second the comment from the original post which said that they didn’t pay much attention in the latter half.
Is there any chance that the human artists experimented with AI and posted the result along with their usual work? :)
That avocado and tomato post took me out, that and the Legos. Very impressive.
The most obvious ai one for me was the last cat picture, somehow it just screamed ai
Huh, I felt the 12/20 was a bit low but I guess not so much. As someone that has never used an image generator (or an LLM for that matter, chatGPT not even once baby) nor has actually worked at tech (though I have been learning programming on my own) and doesn’t even know how to draw… I guess I didn’t do too bad.
Are you proud you haven’t used chatgpt or LLMs or something? They’re incredibly powerful tools, you will fall behind your peers if you don’t learn to use them when appropriate.
Not proud nor ashamed, and you seems to imply LLMs are needed in all human fields
8/20 and I use it everyday. I got all humans correct, but the landscape and stylistic images is what I got wrong. Ai is getting good.
AI just has a niche. Certain things, like highly stylized images, it’s great at because they’re generic, and genetic/common is what AI does best.
No shot the bottom left was ai gen without human help. AI has so much trouble delivering words and text.
There are newer AI tools that can do text accurately. Usually it’s with text provided in the prompt though, so it’s arguably not AI generated, just AI placed.
The unreadable AI text you’re familiar with is done without the AI really knowing what text is, and they’re just making something that looks vaguely like it. It’s the same way they normally handle any kind of object or shape. Newer tools are built specifically for text, so it actually is readable and makes sense.
To me 4 was pretty obvious as a lot of the lines made no sense, such as the ones right below her face.
Also, is there a correlation between image generator and accuracy?
Luckily we’ve already begun developing tools to help detect AI Images with higher accuracy.
Unluckily, they’re also detecting human-made images as AI images.
I said “with higher accuracy than a human curator.” You didn’t really build upon that, no offence. You also didn’t upvote despite literally repeating something that I said. You just like to take up space in people’s inboxes? I’m trying not to be an asshole about it but I feel legitimate confusion about the purpose of your reply.
This isn’t possible as of now, at least not reliably. Yes, you can tailor a model to one specific generative model, but because we have no reliable outlier detection (to train the “AI made detector”), a generative model can always be trained with the detector model incorporated in the training process. The generative model (or a new model only designed to perturb output of the “original” generative model) would then learn to create outliers to the outlier detector, effectively fooling the detector. An outlier is everything that pretends to be “normal” but isn’t.
In short: as of now we have no way to effectively and reliably defend against adversarial examples. This implies, that we have no way to effectively and reliably detect AI generated content.
Please correct me if I’m wrong, I might be mixing up some things.
It already exists, the human accuracy was only 48% average in this study. It’s really easy to beat.
yokonzo@lemmy.world 1 year ago
One thing I’m not sure if it skews anything, but technically ai images are curated more than anything, you take a few prompts, throw it into a black box and spit out a couple, refine, throw it back in, and repeat. So I don’t know if its fair to say people are getting fooled by ai generated images rather than ai curated
logicbomb@lemmy.world 1 year ago
Well, it does say “AI Generated”, which is what they are.
All of the images in the survey were either generated by AI and then curated by humans, or they were generated by humans and then curated by humans.
I imagine that you could also train an AI to select which images to present to a group of test subjects. Then, you could do a survey that has AI generated images that were curated by an AI, and compare them to human generated images that were curated by an AI.
deweydecibel@lemmy.world 1 year ago
Unless they explained that to the participants, it defeats the point of the question.
Zeth0s@lemmy.world 1 year ago
But they were generated by AI. It’s a fair definition
yokonzo@lemmy.world 1 year ago
I mean fair, I just think that kind of thing stretches the definition of “fooling people”
BlueBockser@programming.dev 1 year ago
But not all AI generated images can fool people the way this post suggests. In essence this study then has a huge selection bias, which just makes it ubfit for drawing any kind of conclusion.
popcar2@programming.dev 1 year ago
Technically you’re right but the thing about AI image generators is that they make it really easy to mass-produce results. Each one I used in the survey took me only a few minutes, if that. Some images like the cat ones came out great in the first try. If someone wants to curate AI images, it takes little effort.
MysticKetchup@lemmy.world 1 year ago
I think if you consider how people will use it in real life, where they would generate a bunch of images and then choose the one that looks best, this is a fair comparison. That being said, one advantage of this kind of survey is that it involves a lot of random, one-off images. Trying to create an entire gallery of images with a consistent style and no mistakes, or trying to generate something that follows a design spec is going to be much harder than generating a bunch of random images and asking whether or not they’re AI.
dotMonkey@lemmy.world 1 year ago
I think getting a good image from the AI generators is akin to people putting in effort and refining their art rather than putting a bunch of shapes on the page and calling it done