I do think this is one of the places where yacy has it’s strength since yacy is p2p you can retrieve results from other nodes that have indexed the page thus large websites can collectively be indexed by many nodes.

I also don’t remember having problems with the crawler getting blocked but I also didn’t do a lot of crawls I can see this being a problem though.