A Project to Poison LLM Crawlers

Submitted ⁨⁨14⁩ ⁨hours⁩ ago⁩ by ⁨Disillusionist@piefed.world⁩ to ⁨technology@lemmy.world⁩

Website operators are being asked to feed LLM crawlers poisoned data by a project called Poison Fountain.

The project page links to URLs which provide a practically endless stream of poisoned training data. They have determined that this approach is very effective at ultimately sabotaging the quality and accuracy of AI which has been trained on it.

Small quantities of poisoned training data can significantly damage a language model.

The page also gives suggestions on how to put the provided resources to use.

source

Comments

Sort:hotnew top

eru@mouse.chitanda.moe ⁨2⁩ ⁨hours⁩ ago
i would imagine companies would just filter it out

need some more clever way of hiding it or allow it to be self hosted so that it has various urls

source
- GamingChairModel@lemmy.world ⁨1⁩ ⁨hour⁩ ago
  If I am reading this correctly, anyone who wants to use this service can just configure their HTTP server to act as the man in the middle of the request, so that the crawler sees your URL but is retrieving poison fountain content from the poison fountain service.
  
  If so, that means the crawlers wouldn’t be able to filter by URL because the actual handler that responds to the HTTP request doesn’t ever see the canonical URL of the poison fountain.
  
  In other words, the handler is “self hosted” at its own URL while the stream itself comes from the same URL that the crawler never sees.
  
  source
termaxima@slrpnk.net ⁨8⁩ ⁨hours⁩ ago
Been thinking about making one of these too, especially since I have a catchy name : asbestos

source
vacuumflower@lemmy.sdf.org ⁨11⁩ ⁨hours⁩ ago
If, suppose, I were optimistic over this technology, but pessimistic over its current stage of development, I’d expect this to be a cure. It’s a problem they’ll have to solve. A test they’ll have to pass.

If somewhere inside those things someone makes a mechanism building a graph of syllogisms, no kind of poisoned input data will be able to hurt them.

So - this is a good thing, but when people say it’s a rebellion, it’s not.

source
- FlashMobOfOne@lemmy.world ⁨5⁩ ⁨hours⁩ ago
  
  A test they’ll have to pass.
  
  This makes me chuckle, as they invented euphemisms like ‘hallucinations’ because their LLM models can’t do what they promise. Fabulous marketing, but clearly they didn’t do enough testing.
  
  source
  - vacuumflower@lemmy.sdf.org ⁨1⁩ ⁨hour⁩ ago
    I said, in other words, that it doesn’t matter what they do until this problem is solved. So if this is described as some sort of rebellion against AI (or “AI”), then no. At the point where it becomes dangerous technology in itself and not just for economy, it won’t be.
    
    source
  - Bazoogle@lemmy.world ⁨3⁩ ⁨hours⁩ ago
    
    as they invented euphemisms like ‘hallucinations’
    
    Seems like a pretty accurate word to use, no? Could also use fabrication, concoction, phantom, or something else? I think “lie” and its synonyms are not accurate, since that requires intent. Since the LLM does not have intent, it cannot “lie”.
    
    source
    -> View More Comments
- treesapx@lemmy.world ⁨3⁩ ⁨hours⁩ ago
  “You’re not opposing me. All you’ve done is create a problem that will stop me until I have it figured out.” is the description of every struggle between opposing forces, so it’s interesting that you disagree with that.
  
  source
  - vacuumflower@lemmy.sdf.org ⁨1⁩ ⁨hour⁩ ago
    Not really, more like “if I can find a key to the door, I can open it, so engraving a fixed combination for the door lock on the same key doesn’t change much”.
    
    Poisoned data is fundamentally valid data. Concepts of logical connectivity and statements being true or false are something needed to use it.
    
    source
- Disillusionist@piefed.world ⁨11⁩ ⁨hours⁩ ago
  Not all problems may be cured immediately. Battles are rarely won with a single attack. A good thing is not the same as nothing.
  
  source
chunes@lemmy.world ⁨13⁩ ⁨hours⁩ ago

Small quantities of poisoned training data can significantly damage a language model.

Source: trust me bro.

Nightshade tried the same thing and it never worked.

source
- ExLisper@lemmy.curiana.net ⁨12⁩ ⁨hours⁩ ago
  Here’s your source: www.anthropic.com/research/small-samples-poison
  
  source
- homes@piefed.world ⁨12⁩ ⁨hours⁩ ago
  Night shade did work on older models. Neural models adapted to prevent poisoning.
  
  This is a new approach.
  
  source
  - nullroot@lemmy.world ⁨5⁩ ⁨hours⁩ ago
    Ye, nightshade was defeated by a blur and sharpen iirc lol. Still, was a good first step.
    
    source
Lembot_0006@programming.dev ⁨13⁩ ⁨hours⁩ ago
Idiots: This new technology is still quite ineffective. Let’s sabotage it’s improvement!

Imbeciles: Yeah!

source
- Stern@lemmy.world ⁨13⁩ ⁨hours⁩ ago
  Corpos: Don’t steal our stuff! That’s piracy!
  
  Also corpos: Your stuff? My stuff now.
  
  Bootlickers: Oh my god this shoe polish is delicious.
  
  source
  - FauxLiving@lemmy.world ⁨3⁩ ⁨hours⁩ ago
    Person: Says a thing
    
    Person 2, who disagrees with the thing: YOU’RE A BOOTLICKER!
    
    Super convincing. I’m sure you’re going to win people over to your position if you scream loud enough.
    
    source
  - Lembot_0006@programming.dev ⁨13⁩ ⁨hours⁩ ago
    You should select something: whether you like the current copyright system or not. You can’t do both.
    
    source
    -> View More Comments
- Disillusionist@piefed.world ⁨13⁩ ⁨hours⁩ ago
  AI companies could start, I don’t know- maybe asking for permission to scrape a website’s data for training? Or maybe try behaving more ethically in general? Perhaps then they might not risk people poisoning the data that they clearly didn’t agree to being used for training?
  
  source
  - Lembot_0006@programming.dev ⁨13⁩ ⁨hours⁩ ago
    Why should they ask permission to read freely provided data? Nobody’s asking for any permission, but LLM trainers somehow should? And what do you want from them from an ethical standpoint?
    
    source
    -> View More Comments
FaceDeer@fedia.io ⁨13⁩ ⁨hours⁩ ago
Doesn't work, but I guess if it makes people feel better I suppose they can waste their resources doing this.

Modern LLMs aren't trained on just whatever raw data can be scraped off the web any more. They're trained with synthetic data that's prepared by other LLMs and carefully crafted and curated. Folks are still thinking ChatGPT 3 is state of the art here.

source
- KeenFlame@feddit.nu ⁨1⁩ ⁨hour⁩ ago
  Ai devalues datasets when it refines, many resources are aimed towards solving the degradation that occurs when ai trains on ai. Gradients become poor and quality follows
  
  source
  - FaceDeer@fedia.io ⁨53⁩ ⁨minutes⁩ ago
    You're thinking of "model decay", I take it? That's not really a thing in practice.
    
    source
- Taldan@lemmy.world ⁨2⁩ ⁨hours⁩ ago
  Let’s say I believe you. If that’s the case, why are AI companies still scraping everything?
  
  source
  - FaceDeer@fedia.io ⁨2⁩ ⁨hours⁩ ago
    Raw materials to inform the LLMs constructing the synthetic data, most likely. If you want it to be up to date on the news, you need to give it that news.
    
    The point is not that the scraping doesn't happen, it's that the data is already being highly processed and filtered before it gets to the LLM training step. There's a ton of "poison" in that data naturally already. Early LLMs like GPT-3 just swallowed the poison and muddled on, but researchers have learned how much better LLMs can be when trained on cleaner data and so they already take steps to clean it up.
    
    source
- Disillusionist@piefed.world ⁨12⁩ ⁨hours⁩ ago
  From what I’ve heard, the influx of AI data is one of the reasons actual human data is becoming increasingly sought after. AI training AI has the potential to become a sort of digital inbreeding that suffers in areas like originality and other ineffable human qualities that AI still hasn’t quite mastered.
  
  I’ve also heard that this particular approach to poisoning AI is newer and thought to be quite effective, though I can’t personally speak to its efficacy.
  
  source
  - BagOfHeavyStones@piefed.social ⁨10⁩ ⁨hours⁩ ago
    Faults in replication? That can become cancer for humans. AI as well I guess.
    
    source
- XLE@piefed.social ⁨9⁩ ⁨hours⁩ ago
  Do you have any basis for this assumption, FaceDeer?
  
  Based on your pro-AI-leaning comments in this thread, I don’t think people should accept defeatist rhetoric at face value.
  
  source
  - FaceDeer@fedia.io ⁨6⁩ ⁨hours⁩ ago
    A basic Google search for "synthetic data llm training" will give you lots of hits describing how the process goes these days.
    
    Take this as "defeatist" if you wish, as I said it doesn't really matter. In the early days of LLMs when ChatGPT first came out the strategy for training these things was to just dump as much raw data onto them as possible and hope quantity allowed the LLM to figure something out from it, but since then it's been learned that quality is better than quantity and so training data is far more carefully curated these days. Not because there's "poison" in it, just because it results in better LLMs. Filtering out poison will happen as a side effect.
    
    It's like trying to contaminate a city's water supply by peeing in the river upstream of the water treatment plant drawing from it. The water treatment plant is already dealing with all sorts of contaminants anyway.
    
    source
    -> View More Comments
zr0@lemmy.dbzer0.com ⁨9⁩ ⁨hours⁩ ago
This is just stupid^20

source