Comment on I was wrong about robots.txt

• ell1e@leminal.space ⁨1⁩ ⁨day⁩ ago

  arstechnica.com/…/cloudflare-wants-google-to-chan…

  source

  • General_Effort@lemmy.world ⁨1⁩ ⁨day⁩ ago

    Ok. That quotes a tweet by Cloudflare’s CEO. IDK what his qualifications are, but his conflict of interest is obvious enough. Real quality journalism there.

    Here’s Google technical documentation on its crawlers: developers.google.com/…/google-common-crawlers

    source

    • ell1e@leminal.space ⁨1⁩ ⁨day⁩ ago

      So what’s the quote from your documentation that backs up your claim?

      source

      • General_Effort@lemmy.world ⁨1⁩ ⁨day⁩ ago

        I’m not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?

        source

        • -> View More Comments
• cecilkorik@lemmy.ca ⁨1⁩ ⁨day⁩ ago

  Absolutely true. They’ll buy the data they want from some shitty crawler running from some data broker in some far-flung and lawless part of the world, hallucinate the actual source, and pretend they had no idea their “data partner” wasn’t respecting robots.txt if they have to, which they won’t ever have to do because it’s literally impossible to detect and prove and realistically unenforceable.

  This is a company that removed it’s company motto of “Don’t be evil” because it found it too “limiting”. Don’t be naive.

  source

  • General_Effort@lemmy.world ⁨1⁩ ⁨day⁩ ago

    That’s very different from what I called false.

    What you describe may happen, but probably not as much as you think. Much of that stuff is just not that valuable. Some personal, colloquial writing is necessary, but Google already pays Reddit. Other stuff is better obtained from torrents or shadow libraries like Anna’s Archive.

    source