Comment on Have clankers visited my blog one hundred twenty-one sexagintillion eight hundred ten novemquinquagintillion times so far in November??

carrylex@lemmy.world ⁨1⁩ ⁨week⁩ ago

• Cort@lemmy.world ⁨1⁩ ⁨week⁩ ago

  Can you just turn the robots.txt into a click wrap agreement to charge robots high fees for access above a certain threshold?

  source

  • carrylex@lemmy.world ⁨1⁩ ⁨week⁩ ago

    why do a agreement when you can serve a zip bomb :D

    source
  • Valmond@lemmy.world ⁨1⁩ ⁨week⁩ ago

    Puts the full EU regulations in robot.txt

    source