Comment on Reddit blocking all major search engines, except Google

leopold@lemmy.kde.social ⁨1⁩ ⁨month⁩ ago

• gedaliyah@lemmy.world ⁨1⁩ ⁨month⁩ ago

  "We always obey the robots.txt"

  • A bunch of corporations that have no accountability and plenty of incentive to just ignore it and have all been caught training AI on off-limits data.

  source
• capital@lemmy.world ⁨1⁩ ⁨month⁩ ago

  Rate limiting could “fix” that unfortunately.

  source
• LodeMike@lemmy.today ⁨1⁩ ⁨month⁩ ago

  They’re likely blocking user agents too, which I think also doesn’t have legal enforcement (as in DuckDuckGo can just use “Google” unless they said otherwise.

  source

  • Natanael@slrpnk.net ⁨1⁩ ⁨month⁩ ago

    LinkedIn tried blocking scraping that way but as the scraping isn’t burdensome it’s basically legal but you can still be bound by TOS and civil claims

    natlawreview.com/…/hiq-and-linkedin-reach-propose…

    source