Comment on AI companies are violating a basic social contract of the web and and ignoring robots.txt

KillingTimeItself@lemmy.dbzer0.com ⁨8⁩ ⁨months⁩ ago

• ricdeh@lemmy.world ⁨8⁩ ⁨months⁩ ago

  You cannot simply block crawlers lol

  source

  • bigMouthCommie@kolektiva.social ⁨8⁩ ⁨months⁩ ago

    hide a link no one would ever click. if an ip requests the link, it's a ban

    source

    • T156@lemmy.world ⁨8⁩ ⁨months⁩ ago

      Except that it’d also catch out people who use accessibility devices might see the link anyways, or use the keyboard to navigate a site instead of a mouse.

      source

      • bigMouthCommie@kolektiva.social ⁨8⁩ ⁨months⁩ ago

        i don't know, maybe there's a canvas trick. i'm not a webdev so i am a bit out of my depth and mostly guessing and remembering 20-year-old technology

        source
    • oatscoop@midwest.social ⁨8⁩ ⁨months⁩ ago

      If it weren’t so difficult and require so much effort, I’d rather poison their data.

      Clicking the link causes the server to switch to page content designed to fuck with a LLM.

      source

      • HelloHotel@lemm.ee ⁨8⁩ ⁨months⁩ ago

        Visiting /enter_spoopmode.html will choose a theme and mangle the text accordingly (think search&replace with swear words or santa clause)

        It will also show a banner letting the user know they are in spoop mode, with a javascript button to exit the mode, where the AJAX request URL is ofuscated (think base64) The banner is at the bottom of the html document (not nesisarly the screen itself) and/or inside unusual/normally ignored tags. <script type=“spoop/text” style='display:block">you are in spoop mode</script>

        source
      • T156@lemmy.world ⁨8⁩ ⁨months⁩ ago

        Would that be effective? A lot of poisoning seems targeted to a specific version of an LLM, rather than being general.

        Like how the image poisoning programs only work for some LLMs and not others.

        source

        • -> View More Comments
  • echodot@feddit.uk ⁨8⁩ ⁨months⁩ ago

    Well you can if you know the IPs that come in from but that’s of course the trick.

    source
  • KillingTimeItself@lemmy.dbzer0.com ⁨8⁩ ⁨months⁩ ago

    last i checked humans dont access every page on a website nearly simultaneously…

    And if you imitate a human then honestly who cares.

    source
• Bogasse@lemmy.ml ⁨8⁩ ⁨months⁩ ago

  Detecting crawlers can be easier said than done 🙁

  source

  • KillingTimeItself@lemmy.dbzer0.com ⁨8⁩ ⁨months⁩ ago

    i mean yeah, but at a certain point you just have to accept that it’s going to be crawled. The obviously negligent ones are easy to block.

    source
• KingThrillgore@lemmy.ml ⁨8⁩ ⁨months⁩ ago

  There are more crawlers than I have fucks to give, you’ll be in a pissing match forever. robots.txt was supposed to be the norm to tell crawlers what they can and cannot access. Its not on you to block them. Its on them, and its sadly a legislative issues at this point.

  source

  • KillingTimeItself@lemmy.dbzer0.com ⁨8⁩ ⁨months⁩ ago

    yes but also there’s a point where it’s blatantly obvious. And i can’t imagine it’s hard to get rid of the obviously offending ones. Respectful crawlers are going to be imitating humans, so who cares, disrespectful crawlers will ddos your site, that can’t be that hard to implement.

    Though if we’re talking “hey please dont scrape this particular data” Yeah nobody was ever respecting that lol.

    source

    • Reddfugee42@lemmy.world ⁨8⁩ ⁨months⁩ ago

      Both paragraphs demonstrate gross ignorance

      source