Comment on AI companies are violating a basic social contract of the web and and ignoring robots.txt
ShitpostCentral@lemmy.world 9 months agoYou’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed.
GenderNeutralBro@lemmy.sdf.org 9 months ago
You’d probably have to go out of your way to avoid logging this. I’ve always seen such logs enabled by default when setting up web servers.