Comment on I was wrong about robots.txt

• General_Effort@lemmy.world ⁨1⁩ ⁨month⁩ ago

  I’m not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?

  source

  • ell1e@leminal.space ⁨1⁩ ⁨month⁩ ago

    Nothing on this page seems to contradict the article.

    source

    • General_Effort@lemmy.world ⁨1⁩ ⁨month⁩ ago

      You look up what Googlebot does. No AI.

      You want to know what crawlers do AI? Just search for “AI”, or “training”, or some such, or skim through. It’s not long. Google-Extended collects training data. Note that Google-Extended is explicitly not used to rank pages.

      Did that help?

      source

      • ell1e@leminal.space ⁨1⁩ ⁨month⁩ ago

        You look up what Googlebot does. No AI.

        I disagree that it says that. The Cloudflare CEO seems to disagree as well.

        source

        • -> View More Comments