Millions of websites to get 'game-changing' AI bot blocker

Comments

• Scrollone@feddit.it ⁨9⁩ ⁨months⁩ ago

  I wish there was an alternative (possibly European) to Cloudflare, because it’s so scary to put all eggs in one basket.

  source
• themurphy@lemmy.ml ⁨9⁩ ⁨months⁩ ago

  I didnt read “bot blocker” wrong, thats for sure…

  source
• isVeryLoud@lemmy.ca ⁨9⁩ ⁨months⁩ ago

  So… Proprietary Anubis?

  source
• vermaterc@lemmy.ml ⁨9⁩ ⁨months⁩ ago

  To that end the company is developing a “Pay Per Crawl” system, which would give content creators the option to request payment from AI companies for utilising their original content.

  So Cloudflare is not as much “saving the Internet”, as just becoming a middleman between LLM training companies and content creators. Which I believe has a potential of being a true goldmine in the future.

  source

  • interdimensionalmeme@lemmy.ml ⁨9⁩ ⁨months⁩ ago

    Can you DRM a crawl ?

    source

    • vane@lemmy.world ⁨9⁩ ⁨months⁩ ago

      Oh yes DRM whole internet and wire it to Personal ID. Wet dream.

      source
    • boonhet@sopuli.xyz ⁨9⁩ ⁨months⁩ ago

      You can if you’re Cloudflare.

      source
  • sbv@sh.itjust.works ⁨9⁩ ⁨months⁩ ago

    Corps are gonna corp.

    source
• chunes@lemmy.world ⁨9⁩ ⁨months⁩ ago

  I can’t wait to be denied access to websites because of it. Even more than I already am, that is.

  source
• yesman@lemmy.world ⁨9⁩ ⁨months⁩ ago

  This is not about stopping bot-scrapers, it’s about charging them.

  source

  • spankmonkey@lemmy.world ⁨9⁩ ⁨months⁩ ago

    Hopefully people will price their content out of reach of the bot-scrapers, effectively stopping them.

    source
• Imgonnatrythis@sh.itjust.works ⁨9⁩ ⁨months⁩ ago

  I really wish the answer was a legally enforced robots.txt file that very easily allowed any web data any organization or individual user is posting to script out what the permissions are. I often use a LLM as a search and most of the time the citations are pretty decent and I use those to link out to source content. I run a small blog and I’d love to get indexed in a LLM, not blocked, as long as I was assured a reference link for any content used and had some legal recourse if I found my data was being misused. I don’t love the answer being another mega corporation posing as a white knight looking to skim some money off of the “loophole” that is AI copyright infringement.

  source

  • drmoose@lemmy.world ⁨9⁩ ⁨months⁩ ago

    How would you legally enforce robots.txt? It’s not a legally sound system.

    source
• Concave1142@lemmy.world ⁨9⁩ ⁨months⁩ ago

  Until the AI companies find a way around it. Love the idea so hopefully it causes at least 3 days of struggle for the AI crawlers.

  Having said that… Can someone else put this in place so we do not have Cloudflare hosting everything where we would just be one intern away from a global outage. Please? Pretty please?

  source

  • auraithx@piefed.social ⁨9⁩ ⁨months⁩ ago

    Yeah this will have absolutely no impact to gathering training data.

    I assumed it was to block ai agents crawling it during requests, which they’d be unlikely to bypass in the web ui.

    But no company spending millions on training will hesitate to have an agent appear as a regular desktop user to scrape data.

    source

    • boonhet@sopuli.xyz ⁨9⁩ ⁨months⁩ ago

      Does cloudflare still look at the agent? I thought they have more reliable data points.

      source

      • -> View More Comments
  • WreckingBANG@lemmy.ml ⁨9⁩ ⁨months⁩ ago

    github.com/TecharoHQ/anubis

    source
  • baduhai@sopuli.xyz ⁨9⁩ ⁨months⁩ ago

    Proof of work seems to be working pretty well for many websites.

    source
  • orclev@lemmy.world ⁨9⁩ ⁨months⁩ ago

    The problem is that the biggest service Cloudflare provides is DDoS protection, and doing that requires that you have more bandwidth available than your attacker. Having enough bandwidth to withstand modern botnet powered DDoS attacks is ridiculously expensive (and it’s also a finite resource, there’s only so much backbone infrastructure). Basically it’s economically infeasible to have multiple companies providing the service Cloudflare does. You might be able to get away with two companies doing so, but it’s unlikely you could manage more than that without some of them starting to go bankrupt.

    source

    • acosmichippo@lemmy.world ⁨9⁩ ⁨months⁩ ago

      when a critical service is not economical for more than one business to do (natural monopoly), that’s when govt should be stepping in.

      source

      • -> View More Comments
    • Kowowow@lemmy.ca ⁨9⁩ ⁨months⁩ ago

      I wonder if it would be a good investment for a country to have their own then down the line expand to sell the same service to others

      source

      • -> View More Comments