Comment on I was wrong about robots.txt

• General_Effort@lemmy.world ⁨2⁩ ⁨days⁩ ago

  You look up what Googlebot does. No AI.

  You want to know what crawlers do AI? Just search for “AI”, or “training”, or some such, or skim through. It’s not long. Google-Extended collects training data. Note that Google-Extended is explicitly not used to rank pages.

  Did that help?

  source

  • ell1e@leminal.space ⁨1⁩ ⁨day⁩ ago

    You look up what Googlebot does. No AI.

    I disagree that it says that. The Cloudflare CEO seems to disagree as well.

    source

    • General_Effort@lemmy.world ⁨1⁩ ⁨day⁩ ago

      It would be a lot to write, if you had to say what something does not do rather than what it does.

      I looked at what the Cloudflare CEO said again. To be fair to him, he is not actually backing you up. He’s saying that Google makes no difference between the AI overview and the other search results. That is true. The AI overview is a search feature. I’m not sure why someone would want their link listed in search but not appear much more prominently in the AI overview.

      source

      • ell1e@leminal.space ⁨1⁩ ⁨day⁩ ago

        But the article later does back it up: “Although Cloudflare singled out Google, other search engines that view AI search features as part of their search products also use the same bots for training as they do for search indexing.”

        source

        • -> View More Comments