It doesn’t quite work that way, since the URL is also the model number/SKU which comes from the manufacturer. I suppose I could write an alias for just that product but it would become rather confusing.
What I did experiment with was temporarily deleting the product altogether for a day or two. (We barely ever sell it. Maybe 1 or 2 units of it a year. This is no great loss in the name of science.) This causes our page to return a 404 when you try to request it. The bots blithely ignored this, and continued attempting to hammer that nonexistent page all the same. Puzzling.
This is far beyond my limited coding experience but I do enjoy a good puzzle. In your opinion do you think it could be some gen AI scraper. Like the gen AI is deciding what page to scrape and cuz its stupid it keeps selecting your page?
Alternatively I wonder if the product page just happens to have an unsual combinination of keywords that the bot is looking for. Maybe its looking for cheap prices of RAM and the page has some keywords related to RAM?
Good luck I hope you are able to get them to start hammering that page.
In my case the pattern appears to be some manner of DDoS botnet, probably not an AI scraper. The request origins are way too widespread and none of them resolve down to anything that’s obviously datacenters or any sort of commercial enterprise. It seems to be a horde of devices in consumer IP ranges that have probably be compromised by some malware package or another, and whoever is controlling it directed it at our site for some reason. It’s possible that some bad actor is using a similar malware/bot farm arrangement to scrape for AI training, but I’d doubt it. It doesn’t fit the pattern from that sort of thing from what I’ve seen.
Anyway, my script’s been playing automated whack-a-mole with their addresses and steadily filtering them all out, and I geoblocked the countries where the largest numbers of offenders were. (“This is a bad practice!” I hear the hue and cry from specific strains of bearded louts on the Internet. That says maybe, but I don’t ship to Brazil or Singapore or India, so I don’t particularly care. If someone insists on connecting through a VPN from one of those regions for some reason, that’s their own lookout.)
They seem to have more or less run out of compromised devices to throw at our server, so now I only see one such request every few minutes rather than hundreds per second. I shudder to think how long my firewall’s block list is by now.
dual_sport_dork@lemmy.world 2 days ago
It doesn’t quite work that way, since the URL is also the model number/SKU which comes from the manufacturer. I suppose I could write an alias for just that product but it would become rather confusing.
What I did experiment with was temporarily deleting the product altogether for a day or two. (We barely ever sell it. Maybe 1 or 2 units of it a year. This is no great loss in the name of science.) This causes our page to return a 404 when you try to request it. The bots blithely ignored this, and continued attempting to hammer that nonexistent page all the same. Puzzling.
DoGeeseSeeGod@lemmy.blahaj.zone 2 days ago
This is far beyond my limited coding experience but I do enjoy a good puzzle. In your opinion do you think it could be some gen AI scraper. Like the gen AI is deciding what page to scrape and cuz its stupid it keeps selecting your page?
Alternatively I wonder if the product page just happens to have an unsual combinination of keywords that the bot is looking for. Maybe its looking for cheap prices of RAM and the page has some keywords related to RAM?
Good luck I hope you are able to get them to start hammering that page.
dual_sport_dork@lemmy.world 2 days ago
In my case the pattern appears to be some manner of DDoS botnet, probably not an AI scraper. The request origins are way too widespread and none of them resolve down to anything that’s obviously datacenters or any sort of commercial enterprise. It seems to be a horde of devices in consumer IP ranges that have probably be compromised by some malware package or another, and whoever is controlling it directed it at our site for some reason. It’s possible that some bad actor is using a similar malware/bot farm arrangement to scrape for AI training, but I’d doubt it. It doesn’t fit the pattern from that sort of thing from what I’ve seen.
Anyway, my script’s been playing automated whack-a-mole with their addresses and steadily filtering them all out, and I geoblocked the countries where the largest numbers of offenders were. (“This is a bad practice!” I hear the hue and cry from specific strains of bearded louts on the Internet. That says maybe, but I don’t ship to Brazil or Singapore or India, so I don’t particularly care. If someone insists on connecting through a VPN from one of those regions for some reason, that’s their own lookout.)
They seem to have more or less run out of compromised devices to throw at our server, so now I only see one such request every few minutes rather than hundreds per second. I shudder to think how long my firewall’s block list is by now.