dual_sport_dork@lemmy.world 2 days ago
I run an ecommerce site and lately they’ve latched onto one very specific product with attempts to hammer its page and any of those branching from it for no readily identifiable reason, at the rate of several hundred times every second. I found out pretty quickly, because suddenly our view stats for that page in particular rocketed into the millions.
I had to insert a little script to IP ban these fuckers, which kicks in if I see a malformed user agent string or if you try to hit this page specifically more than 100 times. Through this I discovered that the requests are coming from hundreds of thousands of individual random IP addresses, many of which are located in Singapore, Brazil, and India, and mostly resolve down into those owned by local ISPs and cell phone carriers.
Of course they ignore your robots.txt as well. This smells like some kind of botnet thing to me.
panda_abyss@lemmy.ca 2 days ago
I don’t really get those bots.
Like, there are bots that are trying to scrape product info, or prices, or scan for quantity fields. But why the hell do some of these bots behave the way they do?
Do you use Shopify by chance? With Shopify the bots could be scraping the product.json endpoint unless it’s disabled in your theme. Shopify just seems to show the updated at timestamp from the db in their headers+product data, so inventory quantity changes actually result in a timestamp change that can be used to estimate your sales.
There are companies that do that and sell sales numbers to competitors.
No idea why they have inventory info on their products table, it’s probably a performance optimization.
I haven’t really done much scraping work in a while, not since before these new stupid scrapers started proliferating.
dual_sport_dork@lemmy.world 2 days ago
Negative. Our solution is completely home grown. All artisinal-like, from scratch. I can’t imagine I reveal anything anyone would care about much except product specs, and our inventory and pricing really doesn’t change very frequently.
Even so, you think someone bothering to run a botnet to hound our site would distribute page loads across all of our products, right? Not just one. It’s nonsensical.
panda_abyss@lemmy.ca 2 days ago
Yeah, that’s the kind of weird shit I don’t understand. Someone on the other hand is paying for servers and a residential proxy to send that traffic too. Why?
Nighed@feddit.uk 2 days ago
Can you just move that product to a new URL? What happens?
dual_sport_dork@lemmy.world 2 days ago
It doesn’t quite work that way, since the URL is also the model number/SKU which comes from the manufacturer. I suppose I could write an alias for just that product but it would become rather confusing.
What I did experiment with was temporarily deleting the product altogether for a day or two. (We barely ever sell it. Maybe 1 or 2 units of it a year. This is no great loss in the name of science.) This causes our page to return a 404 when you try to request it. The bots blithely ignored this, and continued attempting to hammer that nonexistent page all the same. Puzzling.
lka1988@lemmy.dbzer0.com 2 days ago
Could it be a competitor for that particular product? Hired some foreign entity to hit anything related to their own product?
dual_sport_dork@lemmy.world 2 days ago
Maybe, but I also carry literally hundreds of other products from that same brand including several that are basically identical with trivial differences, and they’re only picking on that one particular SKU.
porcoesphino@mander.xyz 2 days ago
Have you ever tried writing a scrapper. I have for offline reference material. You’ll do that a few times and know. I usually only want a relatively small site (say a Khan Academy lesson which doesn’t save text offline, just videos) and put in a large delay between requests but I’ll still come back after thinking I have it down and it’s thrashed something