Comment on Is this a good first homeserver?
hirihit640@sh.itjust.works 4 days agoUnfortunate but understandable. I believe there are public lists of the IPs of major VPN providers if you wanted to make an exception, but as VPNs are sometimes used for scraping too, I can see why you might not want that.
Yea so the same server and IP addresses VPN provider use (which are again just cloud providers) are also used for other purposes mainly by scrapers and DDos attacks indeed. Too bad all those cloud providers do not act accordingly in the past 5 years. Since its getting worse and worse. With the increase of datacenters and centralization, I consider it a duty of these companies to take action to stop these scammers, spammers, scrapers, and attackers.
However, that is often not the case. Now I must say, I created Angie Guardian myself (alternative to Anubis). So hopefully soon I can slowly open some of ASN bans. And see how it goes.
hirihit640@sh.itjust.works 3 days ago
Do you know if the scrapers/bots are using the VPN providers? Or if they are just using VPSes in the same datacenter as the VPN, and the datacenter is just NATing all egress traffic to have the same IPv4. In that case I wonder if IPv6 could help distinguish between VPN traffic and bot traffic…
Though ultimately a gate like Anubis or Angie Guardian might be the best solution here so nice work.
The scrapers most likely don't use the VPN providers AFAIK. But the datacenter clusters, so there are dozens of cloud providers where you can rent your own virtual server and those datacenters are also often used by VPSes.
Its actually often coming from separate IP address. Which makes it actually even harder, its not just 1 or 2 IP addresses. Meaning both good and bad traffic has 100.000's of unique IP addresses. At that point I block the whole ASN: https://www.cloudflare.com/learning/network-layer/what-is-an-autonomous-system/
Meaning I basically block the whole datacenter and mark them as bad actors. Cloud server providers F*K up the internet, since before you can easily block one, two or ten IPs and be done. Today, those bad actors and scrapers or DDos attackers can use millions of unique IP addresses, because it can rely on the whole cloud provider IP ranges (ipv6 is even more insane ofc).
hirihit640@sh.itjust.works 2 days ago
Ok that makes sense. As a user I’ve always wondered what it was like from the providers perspective. I always hear people complaining that more and more websites block VPNs, but sounds like it might be better to say more and more websites block scrapers and the VPNs are collateral. Thanks for the insight