Comment on Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal

<- View Parent
tal@lemmy.today ⁨3⁩ ⁨months⁩ ago

Older results will still show up, but these search engines are no longer able to “crawl” Reddit, meaning that Google is the only search engine that will turn up results from Reddit going forward.

Robots.txt lets you ask specific user-agents not to index the site. My guess is that that’s how they restricted it. I don’t know how those changes are reflected in existing indexed pages – don’t know if there’s any standard there – but it’ll stop crawlers from examining new pages.

source
Sort:hotnewtop