Comment on I was wrong about robots.txt
arstechnica.com/…/cloudflare-wants-google-to-chan…
Ok. That quotes a tweet by Cloudflare’s CEO. IDK what his qualifications are, but his conflict of interest is obvious enough. Real quality journalism there.
Here’s Google technical documentation on its crawlers: developers.google.com/…/google-common-crawlers
So what’s the quote from your documentation that backs up your claim?
I’m not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?
Nothing on this page seems to contradict the article.
General_Effort@lemmy.world 1 month ago
Ok. That quotes a tweet by Cloudflare’s CEO. IDK what his qualifications are, but his conflict of interest is obvious enough. Real quality journalism there.
Here’s Google technical documentation on its crawlers: developers.google.com/…/google-common-crawlers
ell1e@leminal.space 1 month ago
So what’s the quote from your documentation that backs up your claim?
General_Effort@lemmy.world 1 month ago
I’m not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?
ell1e@leminal.space 1 month ago
Nothing on this page seems to contradict the article.