Comment on I was wrong about robots.txt
General_Effort@lemmy.world 1 day agoOk. That quotes a tweet by Cloudflare’s CEO. IDK what his qualifications are, but his conflict of interest is obvious enough. Real quality journalism there.
Here’s Google technical documentation on its crawlers: developers.google.com/…/google-common-crawlers
ell1e@leminal.space 1 day ago
So what’s the quote from your documentation that backs up your claim?
General_Effort@lemmy.world 1 day ago
I’m not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?
ell1e@leminal.space 1 day ago
Nothing on this page seems to contradict the article.
General_Effort@lemmy.world 1 day ago
You look up what Googlebot does. No AI.
You want to know what crawlers do AI? Just search for “AI”, or “training”, or some such, or skim through. It’s not long. Google-Extended collects training data. Note that Google-Extended is explicitly not used to rank pages.
Did that help?