And I’m assuming if the robots.txt state their UserAgent isn’t allowed to crawl, it obeys it, right? :P
Comment on Perplexity AI is complaining their plagiarism bot machine cannot bypass Cloudflare's firewall
Kissaki@feddit.org 2 months ago
So, I assume Perplexity uses appropriate identifiable user-agent headers, to allow hosters to decide whether to serve them one way or another?
ubergeek@lemmy.today 2 months ago
Kissaki@feddit.org 2 months ago
No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.
ubergeek@lemmy.today 2 months ago
Except, it’s not a live user hitting 10 sights all the same time, trying to crawl the entire site… Live users cannot do that.
That said, if my robots.txt forbids them from hitting my site, as a proxy, they obey that, right?
drmoose@lemmy.world 2 months ago
Its not up to the hoster to decide whom to serve content. Web is intended to be user agent agnostic.
lime@feddit.nu 2 months ago
yeah it’s almost like there as already a system for this in place
seraphine@lemmy.blahaj.zone 2 months ago
THE CAKE DAY IS NOW. (i dont have an image at hand)
lime@feddit.nu 2 months ago
i really wish we wouldn’t do those. feels too reddity.
seraphine@lemmy.blahaj.zone 2 months ago
as you wish