Comment on A big announcement from Skill Up (new website)
pirateKaiser@sh.itjust.works 3 days agoAs someone who works for a paywallled website, that’s hardly a deterrent. If the site is important enough, they will pay for accounts and crawl until the server melts
HeyJoe@lemmy.world 3 days ago
Is there any true way to block it? Does the crawler literally use the same access (443) as us to scrape content? If so, the only other thing I can think of is to block all known IP’s that AI crap originates from, but that sounds daunting and impossible to catch everything.
pirateKaiser@sh.itjust.works 2 days ago
There’s no fullproof way. Even if you somehow block every crawling automation, there’s still puppeteering where the bot behaves just like a normal user.