Comment on Black Mirror AI
It’s naive to assume that google crawlers respect robot.txt.
It’d be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don’t stay away. 😂
rosco385@lemm.ee 5 weeks ago
It’d be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don’t stay away. 😂