I have an idea. Why don’t I put a bunch of my website stuff in one place, say a pdf, and you screw heads just buy that? We’ll call it a “book”
Cloudflare plans marketplace to sell permission to scrape websites
Submitted 1 month ago by Appoxo@lemmy.dbzer0.com to technology@lemmy.world
Comments
scarabine@lemmynsfw.com 1 month ago
umami_wasbi@lemmy.ml 1 month ago
How can I do this without Cloudflare?
rikudou@lemmings.world 1 month ago
Put a page on your website saying that scrapping your website costs [insert amount] and block the bots otherwise.
gravitas_deficiency@sh.itjust.works 1 month ago
The hard part is reliably detecting the bots
gaylord_fartmaster@lemmy.world 1 month ago
They’re already ignoring robots.txt, so I’m not sure why anyone would think they won’t just ignore this too. All they have to do is get a new IP and change their useragent.
redditReallySucks@lemmy.dbzer0.com 1 month ago
Cloudflare is protecting a lot of sites from scraping with their POW captchas. They could allow people who pay