Comment on A Project to Poison LLM Crawlers

• arcterus@piefed.blahaj.zone ⁨2⁩ ⁨months⁩ ago

  This issue is largely manifesting through AI scraping rjght now. Additionally, many intentionally ignore robots.txt. Currently, LLM scrapers are basically just bad actors on the internet. Courts have also ruled in favor of a number of AI companies when sued in the US, so it’s unlikely anything will change. Effectively, if you don’t like the status quo, stuff like this is one of your few options.

  This isn’t even mentioning of course whether we actually want these companies to improve their models before resolving the problems of energy consumption and potential displacement of human workers.

  source

  • Lembot_0006@programming.dev ⁨2⁩ ⁨months⁩ ago

    All crawlers ignore robots text since the very start. Anyway, if THAT is the problem then IT is a problem, not the LLMs as a whole.

    source

    • FauxLiving@lemmy.world ⁨2⁩ ⁨months⁩ ago

      You can tell when you’re talking with someone who has been given the position of ‘AI Bad’, but doesn’t actually understand the moral positions or technological details that form the foundation of that argument by how confidently they repeat some detail that is clearly nonsense to anybody with knowledge of the subject.

      source
    • arcterus@piefed.blahaj.zone ⁨2⁩ ⁨months⁩ ago

      If this were true (which is nearly impossible since you said “all"), stuff like Anubis wouldn’t exist since you could just toss up a crowd-sourced robots.txt and be done with it.

      source

      • Lembot_0006@programming.dev ⁨2⁩ ⁨months⁩ ago

        I don’t think the word “ignore” means what you think it means :)

        source