Comment on This website is for humans

aarmea@lemmy.world ⁨2⁩ ⁨months⁩ ago

User-Agent: *
Allow: /

• trk@aussie.zone ⁨2⁩ ⁨months⁩ ago

  Is there a more ignored file on the internet than robots.txt?

  source

  • ohshit604@sh.itjust.works ⁨2⁩ ⁨months⁩ ago

    Do dnt headers count?

    source