And I’m assuming if the robots.txt state their UserAgent isn’t allowed to crawl, it obeys it, right? :P
Comment on Perplexity AI is complaining their plagiarism bot machine cannot bypass Cloudflare's firewall
Kissaki@feddit.org 3 weeks ago
So, I assume Perplexity uses appropriate identifiable user-agent headers, to allow hosters to decide whether to serve them one way or another?
ubergeek@lemmy.today 3 weeks ago
Kissaki@feddit.org 3 weeks ago
No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.
ubergeek@lemmy.today 3 weeks ago
Except, it’s not a live user hitting 10 sights all the same time, trying to crawl the entire site… Live users cannot do that.
That said, if my robots.txt forbids them from hitting my site, as a proxy, they obey that, right?
drmoose@lemmy.world 3 weeks ago
Its not up to the hoster to decide whom to serve content. Web is intended to be user agent agnostic.
lime@feddit.nu 3 weeks ago
yeah it’s almost like there as already a system for this in place
seraphine@lemmy.blahaj.zone 3 weeks ago
THE CAKE DAY IS NOW. (i dont have an image at hand)
lime@feddit.nu 3 weeks ago
i really wish we wouldn’t do those. feels too reddity.
seraphine@lemmy.blahaj.zone 3 weeks ago
as you wish