Comment on This website is for humans

aarmea@lemmy.world ⁨1⁩ ⁨day⁩ ago

User-Agent: *
Allow: /

• trk@aussie.zone ⁨1⁩ ⁨day⁩ ago

  Is there a more ignored file on the internet than robots.txt?

  source

  • ohshit604@sh.itjust.works ⁨1⁩ ⁨day⁩ ago

    Do dnt headers count?

    source