Google Using Generic & Undocumented Google Crawler

Submitted ⁨⁨1⁩ ⁨week⁩ ago⁩ by ⁨tonytins@pawb.social⁩ to ⁨technology@lemmy.world⁩

Comments

Sort:hotnew top

LodeMike@lemmy.today ⁨1⁩ ⁨week⁩ ago

The crawler user agent is just named “Google,”

source
- skeezix@lemmy.world ⁨1⁩ ⁨week⁩ ago
  Very sneaky.
  
  source
Flyswat@lemmy.dbzer0.com ⁨1⁩ ⁨week⁩ ago
Can we get rid of the current model where websites are financed by ads and have instead these AI companies pay the websites they crawl?

I could really do with less ads if they are going to crawl the web anyway.

source
Toes@ani.social ⁨1⁩ ⁨week⁩ ago
Probably just a misconfigured bot and nothing special.

source