This new data poisoning tool lets artists fight back against generative AI

Submitted ⁨⁨1⁩ ⁨year⁩ ago⁩ by ⁨ElectroVagrant@lemmy.world⁩ to ⁨technology@lemmy.world⁩

https://www.technologyreview.com/2023/10/23/1082189/data-poisoning-artists-fight-generative-ai/

A new tool lets artists add invisible changes to the pixels in their art before they upload it online so that if it’s scraped into an AI training set, it can cause the resulting model to break in chaotic and unpredictable ways.

The tool, called Nightshade, is intended as a way to fight back against AI companies that use artists’ work to train their models without the creator’s permission.
[…]
Zhao’s team also developed Glaze, a tool that allows artists to “mask” their own personal style to prevent it from being scraped by AI companies. It works in a similar way to Nightshade: by changing the pixels of images in subtle ways that are invisible to the human eye but manipulate machine-learning models to interpret the image as something different from what it actually shows.

source

Comments

Sort:hotnew top

MargotRobbie@lemmy.world ⁨1⁩ ⁨year⁩ ago
It’s made by Ben Zhao? You mean the “anti AI plagerism” UChicago professor who illegally stole GPLv3 code from an open source program called DiffusionBee for his proprietary Glaze software (reddit link), and when pressed, only released the code for the “front end” while still being in violation of GPL?

The Glaze tool that promised to be invisible to the naked eyes, but contained obvious AI generated artifacts? The same Glaze that reddit defeated in like a day after release?

Don’t take anything this grifter says seriously, I’m surprised he hasn’t been suspended for academic integrity violation yet.

source
- ElectroVagrant@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Thanks for added background! I haven’t been monitoring this area very closely so wasn’t aware, but I’d have thought a publication that has been would then be more skeptical and at least mention some of this, particularly highlighting disputes over the efficacy of the Glaze software. Not to mention the others they talked to for the article.
  
  Figures that in a space rife with grifters you’d have ones for each side.
  
  source
  - Zeth0s@lemmy.world ⁨1⁩ ⁨year⁩ ago
    People don’t understand AI. Probably all articles I have read on it by mainstream media were somehow wrong. It often feels like reading a political journalist discussing about quantum mechanics.
    
    My rule of thumb is: always assume that the articles on AI are wrong. I know it isn’t nice, but that’s the sad reality. Society is not ready for AI because too many few people understand AI. Even AI creators don’t fully understand AI (this is why you often hear about “emergent abilities” of models, it means “we really didn’t expect it and we don’t understand how this happened”)
    
    source
    -> View More Comments
- p03locke@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago
  
  who illegally stole GPLv3 code from an open source program called DiffusionBee for his proprietary Glaze software (reddit link), and when pressed, only released the code for the “front end” while still being in violation of GPL?
  
  Oh, how I wish the FSF had more of their act together nowadays and were more like the EFF or ACLU.
  
  source
  - MargotRobbie@lemmy.world ⁨1⁩ ⁨year⁩ ago
    You should check out the decompilation they did on Glaze too, apparently it’s hard coded to throw out a fake error upon detecting being ran on an A100 as some sort of anti-adversarial training measure.
    
    source
    -> View More Comments
- Dadifer@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Thank you, Margot Robbie! I’m a big fan!
  
  source
  - MargotRobbie@lemmy.world ⁨1⁩ ⁨year⁩ ago
    You’re welcome. Bet you didn’t know that I’m pretty good at tech too.
    
    Also, that’s Academy Award nominated character actress Margot Robbie to you!
    
    source
lloram239@feddit.de ⁨1⁩ ⁨year⁩ ago
“New snake oil to give artists a false sense of security” - The last of these tools I tried had absolutely zero effect on the AI, which is not exactly surprising given that there are hundreds of different ways to make use of image data as well as lots of completely different models. You’ll never cover that all with some pixel twisting.

source
Blaster_M@lemmy.world ⁨1⁩ ⁨year⁩ ago
Oh no, another complicated way to jpeg an image that an ai training program will be able to just detect and discard in a week’s time.

source
- vidarh@lemmy.stad.social ⁨1⁩ ⁨year⁩ ago
  They don’t even need to detect them - once they are common enough in training datasets the training process will “just” learn that the noise they introduce are not features relevant to the desired output. If there are enough images like that it might eventually generate images with the same features.
  
  source
MamboGator@lemmy.world ⁨1⁩ ⁨year⁩ ago
This is cool. I think generative AI is great, but the way it’s being trained right now largely without consent from the artists or subjects is unequivocally unethical. Until the law catches up with the technology, people need ways of protecting themselves.

source
- 0xD@infosec.pub ⁨1⁩ ⁨year⁩ ago
  I don’t see a problem with it training on all materials, fuck copyright. I see the problem in it infringing on everyone’s copyright and then being proprietary, monetized bullshit.
  
  If it trains on an open dataset, it must be completely and fully open. Everything else is peak capitalism.
  
  source
  - Smoogs@lemmy.world ⁨1⁩ ⁨year⁩ ago
    You’re not owed nor entitled to an artist’s time and work for free.
    
    source
    -> View More Comments
- 9thSun@midwest.social ⁨1⁩ ⁨year⁩ ago
  How is training AI with art on the web different to a person studying art styles? I’d say if the AI is being monetized in some capacity, then sure maybe there should be laws in place. I’m just hard-pressed to believe that anyone can have sole control of anything once it gets on the Internet.
  
  source
  - Zeth0s@lemmy.world ⁨1⁩ ⁨year⁩ ago
    I work in AI and I believe it is different. Society is built to distribute wealth, so that everyone can live a decent life. People and AI should be treated differently in front of the law. Non-commercial, open source AI should be treated differently than commercial or closed source models
    
    source
    -> View More Comments
  - realharo@lemm.ee ⁨1⁩ ⁨year⁩ ago
    
    How is training AI with art on the web different to a person studying art styles?
    
    Human brains clearly work differently than AI, how is this even a question?
    
    The term “learning” in machine learning is mainly a metaphor.
    
    source
    -> View More Comments
  - FooBarrington@lemmy.world ⁨1⁩ ⁨year⁩ ago
    I agree that the training isn’t fundamentally different, but that monetization of the output has to be controlled. The big difference between AI and humans is the speed with which they create - you have to employ an army of humans to match the output of a couple of GPUs. For noncommercial projects this is amazing. For commercial projects, it destroys the artists livelihoods.
    
    But this simply means that training shouldn’t be controlled, inference in commercial contexts should be.
    
    source
  - rhombus@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
    The real issue comes in ownership of the AI models and the vast amount of labor involved in the training data. It’s taking what is probably hundreds of thousands of hours of labor in the form of art and converting it into a proprietary machine, all without compensating the artists involved. Whether you can make a comparison to a human studying art is irrelevant, because a corporation can’t own an artist, but they can own an AI and not have to pay it.
    
    source
- regbin_@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Disagree. It’s only unethical if you use it to generate the artist’s piece and claim it as yours.
  
  source
  - MamboGator@lemmy.world ⁨1⁩ ⁨year⁩ ago
    [deleted]
    source
    -> View More Comments
- ElectroVagrant@lemmy.world ⁨1⁩ ⁨year⁩ ago
  
  Until the law catches up with the technology, people need ways of protecting themselves.
  
  I agree, and I wonder if the law might be kicked into catching up quicker as more companies try to adopt these tools and inadvertently infringe on other companies’ copyrighted material. 😅
  
  source
egeres@lemmy.world ⁨1⁩ ⁨year⁩ ago
Here’s the paper: arxiv.org/pdf/2302.04222.pdf

I find it very interesting that someone went in this direction to try to find a way to mitigate plagiarism. This is very akin to adversarial attacks in neural networks (you can read more in this short review arxiv.org/pdf/2303.06032.pdf)

I saw some comments saying that you could just build an AI that detects poisoned images, but that wouldn’t be feasible with a simple NN classifier or feature-based approaches. This technique changes the artist style itself to something the AI would see differently in the latent space, yet, visually perceived as the same image. So if you’re changing to a different style the AI has learned, it’s fair to assume it will be realistic and coherent. Although maaaaaaaybe you could detect poisoned images with some dark magic tho, get the targeted AI then analyze the latent space to see if the image has been tampered with

On the other hand, I think if you build more robust features and just scale the data this problems might go away with more regularization in the network. Plus, it assumes you have the target of one AI generation tool, there are a dozen of these, and if someone trains with a few more images in a cluster, that’s it, you shifted the features and the poisoned images are invalid

source
- vidarh@lemmy.stad.social ⁨1⁩ ⁨year⁩ ago
  Trying to detect poisoned images is the wrong approach. Include them in the training set and the training process itself will eventually correct for it.
  
  I think if you build more robust features
  
  Diffusion approaches etc. do not involve any conscious “building” of features in the first place. The features are trained by training the net to match images with text features correctly, and then “just” repeatedly predict how to denoise an image to get closer to a match with the text features. If the input includes poisoned images, so what? It’s no different than e.g. compression artifacts, or noise.
  
  These tools all try to counter models trained without images using them in the training set with at most fine-tuning, but all they show is that models trained without having seen many images using that particular tool will struggle.
  
  But in reality, the massive problem with this is that we’d expect any such tool that becomes widespread to be self-defeating, in that they become a source for images that will work their way into the models at a sufficient volume that the model will learn them. In doing so they will make the models more robust against noise and artifacts, and so make the job harder for the next generation of these tools.
  
  In other words, these tools basically act like a manual adversarial training source, and in the long run the main benefit coming out of them will be that they’ll prod and probe at failure modes of the models and help remove them.
  
  source
  - RubberElectrons@lemmy.world ⁨1⁩ ⁨year⁩ ago
    Just to start with, not very experienced with neural networks at all beyond messing with openCV for my graduation project.
    
    Anyway, that these countermeasures expose “failure modes” in the training isn’t a great reason to stop doing this, e.g. scammers come up with a new technique, we collectively respond with our own countermeasures. From what I understand, the training process itself has a vulnerability to poisoned inputs just because of how long it takes to train on multiple variations of the same datasets.
    
    If the network feedbacks itself, then cool! It has developed its own style, which is fine. The goal is to stop people from outright copying existing artists style.
    
    source
    -> View More Comments
- nandeEbisu@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Haven’t read the paper so not sure about the specifics, but if it relies on subtle changes, would rounding color values or down sampling the image blur that noise away?
  
  source
  - RubberElectrons@lemmy.world ⁨1⁩ ⁨year⁩ ago
    Wondering the same thing. Slight loss of detail but still successfully gets the gist of the original data.
    
    For that matter, how does the poisoning hold up against regular old jpg compression?
    
    source
TheWiseAlaundo@lemmy.whynotdrs.org ⁨1⁩ ⁨year⁩ ago
Lol… I just read the paper, and Dr Zhao actually just wrote a research paper on why it’s actually legally OK to use images to train AI. Hear me out…

He changes the ‘style’ of input images to corrupt the ability of image generators to mimic them, and even shows that the super majority of artists even can’t tell when this happens with his program, Glaze… Style is explicitly not copywriteable in US case law, and so he just provided evidence that the data OpenAI and others use to generate images is transformative which would legally mean that it falls under fair use.

No idea if this would actually get argued in court, but it certainly doesn’t support the idea that these image generators are stealing actual artwork.

source
- Flambo@lemmy.world ⁨1⁩ ⁨year⁩ ago
  So tl;dr he/his team did two things:
  
  argue the way AI uses content to train is legal
  
  provide artists a tool to prevent their content being used to train AI without their permission
  
  On the surface it sounds all good, but I can’t help but notice a future conflict of interest for Zhao should Glaze ever become monetized. If it were to be ruled illegal to train AI on content without permission, tools like Glaze would be essentially anti-theft devices, but while it remains legal to train AI this way, tools like Glaze stand to perhaps become necessary for artists to maintain the pre-AI status quo w/r/t how their work can be used and monetized.
  
  source
leaky_shower_thought@feddit.nl ⁨1⁩ ⁨year⁩ ago
I am sure we already got a budget version of this called the jpeg.

source
- seaQueue@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Speaking of jpeg I miss the “needs more jpeg” bot that used to run on reddit, that shit was hilarious.
  
  source
  - gregorum@lemm.ee ⁨1⁩ ⁨year⁩ ago
    Reddit was Reddit for 18 fucking years. Just abandoning it leaves a massive hole. It’s gonna take a long time to fill it.
    
    :(
    
    source
    -> View More Comments
wizardbeard@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago
This is already a concept in the AI world and is often used while a model is being trained specifically to make it better. I believe it’s called adversarial training or something like that.

source
- Mango@lemmy.world ⁨1⁩ ⁨year⁩ ago
  No, that’s something else entirely. Adversarial training is where you put an ai against a detector AI as a kind of competition for results.
  
  source
- driving_crooner@lemmy.eco.br ⁨1⁩ ⁨year⁩ ago
  Its called adversarial attack, this is an old video (5 years) explaining how it works and how you can potentially do it charging just one pixel on the image.
  
  youtu.be/SA4YEAWVpbk?si=xObPveXTT2ip5ICG
  
  source
Vodik_VDK@lemmy.world ⁨1⁩ ⁨year⁩ ago
New CAPCHA just dropped.

source
Kolanaki@yiffit.net ⁨1⁩ ⁨year⁩ ago
“I can tell this is toxic by the pixels.”

source
- ElectroVagrant@lemmy.world ⁨1⁩ ⁨year⁩ ago
  “We like to call them poison pixels.”
  
  source
afraid_of_zombies@lemmy.world ⁨1⁩ ⁨year⁩ ago
I am waiting for the day that some obsessed person starts finding ways to do like code injection in pictures.

source
- Rootiest@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Here ya go
  
  source
uriel238@lemmy.blahaj.zone ⁨1⁩ ⁨year⁩ ago
I remember in the early 2010s reading an article like this one on openai.com talking about the dangers of using AI for image search engines to moderate against unwanted content. At the time the concern was CSAM salted to prevent its detection (along with other content salted with CSAM to generate false positives).

My guess is since we’re still training AI with pools of data-entry people who tag pictures with what they appear to be, so that AI reads more into images than their human trainers (the proverbial man inside the Iron Turk).

This is going to be an interesting technology war.

source
I_Has_A_Hat@startrek.website ⁨1⁩ ⁨year⁩ ago
Like trying to stop a flood with a roll of paper towels.

source
gsa@kbin.social ⁨1⁩ ⁨year⁩ ago
Sorry "Artists" but I'm still working around your silly tools and generating beautiful AI Art 🥹

source
- the_q@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Hey! It’s the human herpe, gma! He always shows up with the shittiest takes because his mother drank while pregnant!
  
  source
- KeenFlame@feddit.nu ⁨1⁩ ⁨year⁩ ago
  Makes porn
  
  source
guyrocket@kbin.social ⁨1⁩ ⁨year⁩ ago
Invisible changes to pixels sound like pure BS to me. I'm sure others know more about it than i do but I thought pixels were very simple things.

source
- seaQueue@lemmy.world ⁨1⁩ ⁨year⁩ ago
  “Invisible changes to pixels” means “a human can’t tell the difference with a casual glance” - you can still embed a shit-ton of data in an image that doesn’t look visually like it’s been changed without careful inspection of the original and the new image.
  
  If this data is added in certain patterns it will cause ML models trained against the image to draw incorrect conclusions. It’s a technical hurdle that will slow a casual adversary, someone will post a model trained to remove this sometime soon and then we’ll have a good old software arms race and waste a shit ton of greenhouse emissions adding and removing noise and training ever more advanced models to add and remove it.
  
  source
- Unaware7013@kbin.social ⁨1⁩ ⁨year⁩ ago
  
  I'm sure others know more about it than i do but I thought pixels were very simple things.
  
  You're right, in that pixels are very simple things. However, you and I can't tell one pixel from another in an image, and at the scale of modern digital art (my girlfriend does hers at 300dpi), shifting a handful of pixels isn't going to make much of a visible difference to a person, but a LLM will notice them.
  
  source
  - vidarh@lemmy.stad.social ⁨1⁩ ⁨year⁩ ago
    An AI model will “notice them” but ignore them if trained on enough copies with them to learn that they’re not significant.
    
    source
  - ClamDrinker@lemmy.world ⁨1⁩ ⁨year⁩ ago
    LLM is the wrong term. That’s Large Language Model. These are generative image models / text-to-image models.
    
    Trurhfully though, while it will be there when the image is trained, it won’t ‘notice’ it unless you distort it significantly (enough for humans to notice as well). Otherwise it won’t make much of a difference because these models are often trained on a compressed and downsized version of the image (in what’s called latent space)
    
    source
- wheresmypillow@lemmy.one ⁨1⁩ ⁨year⁩ ago
  A pixel has a binary representation. All of the significant bits for the pixel may not not be needed to display the color of that pixel so there is often excess that can be used or modified. A person wouldn’t see it but an AI reading just the binary would.
  
  source
- theneverfox@pawb.social ⁨1⁩ ⁨year⁩ ago
  Pixels are very simple things, literally 3-5 3 digit numbers.
  
  But pixels mean little too a generative AI - it’s all about relationship between pixels. All AI are high dimensional shapes right now… If you break up the shape strategically, it’ll poison the image
  
  Will this poison pill work? Probably, for at least a while…
  
  source
- Narrrz@kbin.social ⁨1⁩ ⁨year⁩ ago
  have you ever seen those composite images made by combining a huge number of other, radically docent images in such a way that each while image acts like one "pixel" of the overall image? i bet AI models 'see' those images very differently than we do.
  
  source
gregorum@lemm.ee ⁨1⁩ ⁨year⁩ ago
Ooo, this is fascinating. It reminds me of that weird face paint that bugs out facial-recognition in CCTV cameras.

source
- seaQueue@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Or the patterned vinyl wraps they used on test cars that interfere with autofocus in most cameras.
  
  source
penix@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
There is probably a trivial workaround to this.

source
- FaceDeer@kbin.social ⁨1⁩ ⁨year⁩ ago
  There's trivial workarounds for Glaze, which this is based off of, so I wouldn't be surprised.
  
  source
- vidarh@lemmy.stad.social ⁨1⁩ ⁨year⁩ ago
  Yes: Train on more images processed by this.
  
  In other words: If the tool becomes popular it will be self-defeating by producing a large corpus of images teaching future models to ignore the noise it introduces.
  
  source
- Meowoem@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
  It doesn’t even need a work around, it’s not going to affect anything when training a model.
  
  It might make style transfer harder using them as reference images on some models but even that’s fairly doubtful, it’s just noise on an image and everything is already full of all sorts of different types of noise.
  
  source
- hh93@lemm.ee ⁨1⁩ ⁨year⁩ ago
  The problem is identifying it. If it’s necessary to preprocess every image used for training instead of just feeding it is a model that already makes it much more resources costly
  
  source
  - vidarh@lemmy.stad.social ⁨1⁩ ⁨year⁩ ago
    You wouldn’t want to. If you just feed it to the models, then if there are enough of these images to matter the model will learn to ignore the differences. You very specifically don’t want to prevent the model from learning to overcome these things, exactly because if you do you’re stuck with workarounds like that forever, but if you don’t the model will just become more robust to noisy data like this.
    
    source
ayaya@lemdro.id ⁨1⁩ ⁨year⁩ ago
Obviously this is using some bug and/or weakness in the existing training process, so couldn’t they just patch the mechanism being exploited?

Or at the very least you could take a bunch of images, purposely poison them, and now you have a set of poisoned images and their non-poisoned counterparts allowing you to train another model to undo it.

Sure you’ve set up a speedbump but this is hardly a solution.

source
- MxM111@kbin.social ⁨1⁩ ⁨year⁩ ago
  Obviously, with so many different AIs, this can not be a factor (a bug).
  
  If you have no problem looking at the image, then AI would not either. After all both you and AI are neural networks.
  
  source
  - skulblaka@kbin.social ⁨1⁩ ⁨year⁩ ago
    The neural network of a human and of an AI operate in fundamentally different ways. They also interact with an image in fundamentally different ways.
    
    source
    -> View More Comments
  - driving_crooner@lemmy.eco.br ⁨1⁩ ⁨year⁩ ago
    An AI don’t see the images like we do, an AI see a matrix of RGB values and the relationship they have with each other and create an statistical model of the color value of each pixel for a determined prompt.
    
    source
    -> View More Comments
- AnonTwo@kbin.social ⁨1⁩ ⁨year⁩ ago
  
  Obviously this is using some bug and/or weakness in the existing training process, so couldn’t they just patch the mechanism being exploited?
  
  I'd assume the issue is that if someone tried to patch it out, it could legally be shown they were disregarding people's copyright.
  
  source
  - FaceDeer@kbin.social ⁨1⁩ ⁨year⁩ ago
    It isn't against copyright to train models on published art.
    
    source
    -> View More Comments
- egeres@lemmy.world ⁨1⁩ ⁨year⁩ ago
  No! It’s not using an internal exploit, it’s rather about finding a way to visually represent almost the same image, but instead using latent features with different artists (e.g, which would confuse a dreambooth+lora training), however, the method they proposed is flawed, I commented more on lemmy.world/comment/4770884
  
  source
BellaDonna@mujico.org ⁨1⁩ ⁨year⁩ ago
What a dumb solution to a problem that doesn’t need a solution. The problem isn’t AI, it’s the lack of understanding for the tech that has people thinking AI is theft.

source
TropicalDingdong@lemmy.world ⁨1⁩ ⁨year⁩ ago
The AI can have some NaN, as a treat…

source
zwaetschgeraeuber@lemmy.world ⁨1⁩ ⁨year⁩ ago
this is so dumb and clear it wont work at all. thats not the slightest how ai trains on images.

you would be able to get around this tool by just doing the nft thing and screenshot the image and boom code in the picture is erased.

source