Comment on Reddit blocking all major search engines, except Google

• generaldenmark@programming.dev ⁨3⁩ ⁨months⁩ ago

  I worked a company that used product data from our competitors (you can debate the morals of it, but everyone is doing it). Our crawlers were set up so that each new line of requests came from a new IP… I don’t recall the name of the service, and it was not that many unique IP’s but it did allow our crawlers to live unhindered…

  We didn’t do IP banning for the same reasoning, but we did notice one of our competitors did not alter their IP when scraping us. If we had malicious intend, we could have changed data around for their IP only. Eg. increasing the prices, or decreasing the prices so they had bad data…

  I’d imagine companies like OpenAI has many times the IP, and they’d be able to do something similarly… meaning if you try’n ban IP’s, you might hit real users as well… which would be unfortunate.

  source
• GenosseFlosse@feddit.org ⁨3⁩ ⁨months⁩ ago

  Blocking bots is hard, because with some work they can be made to look like users, down to simulating curved mouse movements drom one button to the next if you are really ambitious.

  source

  • JovialMicrobial@lemm.ee ⁨3⁩ ⁨months⁩ ago

    So your saying reddit’s activity analytics can’t necessarily tell the difference between human activity and bot activity?

    So the actual number of people using reddit vs bots isn’t very clear. Someone should tell Reddit’s share holders that’s there’s no way to tell if the advertisements are actually being viewed by people, and there’s no way to tell how much the activity reports have been inflated by bots. I bet they wouldn’t like that very much.

    source

    • GenosseFlosse@feddit.org ⁨3⁩ ⁨months⁩ ago

      Always has been. Technically the server sees no difference in what a browser does vs what a bot does: Downloading files and submitting requests.

      source
• recapitated@lemmy.world ⁨3⁩ ⁨months⁩ ago

  We have a variety of tactics and always adding more

  source