They explained the reason in the comment you just replied to.
Comment on Why I recommend against Brave.
Jerry@feddit.online 3 weeks agoI see no legitimate reason for not using a User Agent string, like all the other crawlers use, other than the desire to hide the crawler and make it difficult to block.
I don't accept his explanation. I see it as gaslighting.
Ulrich@feddit.org 3 weeks ago
yqccv9v3li8o@lemmy.sdf.org 3 weeks ago
Why should the crawler be blockable? That only brings disadvantages for a search engine.
unautrenom@jlai.lu 3 weeks ago
It’s not about ‘Google’ vs ‘the other sear h engines’. It’s about transparency. You’ve probably read some news about how AI crawlers have been destroying infrastrucure and half the time does NOT declare themselves as crawlers in their UA.
Can confirm that nealy 90% (read hundred of thousands) of daily visits to several of my websites are made by crawlers from datacenters and I HATE not knowing whose who. Because when I don’t know, I block and report. Website owners already have enough between AI, Page Rankings, and Research Agencies who all exploit free infra for theie own business.
Do I make exception for Search Engine crawlers? Yeah, I do. I’ve seen Google, Bing, and Mojeek, but weirdly enough, never Brave. Now I know why. And frankly, if they can’t be bothered to be transparent about their crawlings, then I won’t be bothered to make exceptions for them. They’re freeloading just as much as the rest. If they act like shady chinese crawlers, then they have no right to go pikachu face when they’re treated like one.
Jerry@feddit.online 3 weeks ago
Well said
Ulrich@feddit.org 3 weeks ago
Brave doesn’t have AI crawlers, they have search index crawlers.