Comment on Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal
Brewchin@lemmy.world 3 months ago
Parts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google?
This needs to be regulated before the Internet becomes like streaming TV.
tal@lemmy.today 3 months ago
Robots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm.
That isn’t to say that the practice of signing agreements isn’t potentially a concern.
reddig33@lemmy.world 3 months ago
What isn’t the norm is to serve one robots.txt to one company, and a different robots.txt to everyone else. Which is what Reddit is doing here.