Comment on Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal
Eril@feddit.org 3 months agoActually currently it contains this:
User-agent: * Disallow: /
Well, that actually is a blanket ban for everyone, so something else must be at play here.
starman@programming.dev 3 months ago
merj.com/…/investigating-reddits-robots-txt-cloak…
Reddit is serving different file to google
russjr08@bitforged.space 3 months ago
That’s real rich, coming from Reddit.