Comment on Why I recommend against Brave.
yqccv9v3li8o@lemmy.sdf.org 1 month agoWhy should the crawler be blockable? That only brings disadvantages for a search engine.
Comment on Why I recommend against Brave.
yqccv9v3li8o@lemmy.sdf.org 1 month agoWhy should the crawler be blockable? That only brings disadvantages for a search engine.
unautrenom@jlai.lu 1 month ago
It’s not about ‘Google’ vs ‘the other sear h engines’. It’s about transparency. You’ve probably read some news about how AI crawlers have been destroying infrastrucure and half the time does NOT declare themselves as crawlers in their UA.
Can confirm that nealy 90% (read hundred of thousands) of daily visits to several of my websites are made by crawlers from datacenters and I HATE not knowing whose who. Because when I don’t know, I block and report. Website owners already have enough between AI, Page Rankings, and Research Agencies who all exploit free infra for theie own business.
Do I make exception for Search Engine crawlers? Yeah, I do. I’ve seen Google, Bing, and Mojeek, but weirdly enough, never Brave. Now I know why. And frankly, if they can’t be bothered to be transparent about their crawlings, then I won’t be bothered to make exceptions for them. They’re freeloading just as much as the rest. If they act like shady chinese crawlers, then they have no right to go pikachu face when they’re treated like one.
Jerry@feddit.online 1 month ago
Well said
Ulrich@feddit.org 1 month ago
Brave doesn’t have AI crawlers, they have search index crawlers.