Comment on Google Search is now using AI to replace headlines
JensSpahnpasta@feddit.org 2 weeks agoYeah, that’s a problem. So even if you want to search your local newspaper, you won’t be able to because of bot protection. Your crawler will be blocked. You also can’t put the whole of Reddit or even Lemmy in your search index.
yourgodlucifer@lemmy.world 2 weeks ago
I do think this is one of the places where yacy has it’s strength since yacy is p2p you can retrieve results from other nodes that have indexed the page thus large websites can collectively be indexed by many nodes.
I also don’t remember having problems with the crawler getting blocked but I also didn’t do a lot of crawls I can see this being a problem though.