Comment on Another massive distributed HTTP flood is currently hitting git.friendi.ca a
utzer@f.utzer.de@f.utzer.de 16 hours ago is there anyone who can help to figure out if this is a huge scraping or a DDoS?
utzer@f.utzer.de@f.utzer.de 16 hours ago
pasjrwoctx@social.2ndshot.photos 16 hours ago
did I make any sense
@pasjrwoctx Thanks, I checked the user agents. The pattern looks highly coordinated.
Out of 164,435 Anubis challenge requests, there were only 160 distinct user-agent strings. The top ten account for 96.79% of all requests and were used almost perfectly evenly:
These are old browser versions from around 2023. Combined with almost one IP per request, this looks like a single coordinated tool rotating through a fixed list of spoofed browser identities and a very large residential/proxy network. It does not prove whether the objective is scraping or disruption, but it is clearly not ordinary independent crawling.
pasjrwoctx@social.2ndshot.photos 16 hours ago
@utzer ok so you could block those user agents I have something like # Fake browser detection
RewriteCond %{HTTP_USER_AGENT} (Chrome/[0-9]{3}|Chrome/1[3-9][0-9]|Chrome/150|Firefox/1[3-9][0-9]|Safari/60[0-9]|Version/17)
[NC]RewriteCond %{HTTP_ACCEPT_LANGUAGE} ^$
RewriteRule ^ - [G,L]
RewriteCond %{THE_REQUEST} "GET\shttp"
[NC]RewriteRule ^ - [G,L] and then # 1. BLOCK BAD BROWSER NAMES / BOT FRAMEWORKS
RewriteCond %{HTTP_USER_AGENT} (CCBot|SearchEngineBot|Pandalytics|UCBrowser|ZoneProjectBot|Embarcadero\sURI\sClient|Xenu\sLink\sSleuth|siteradar|SignalsBot|fun-cert-watch|SERankingBacklinksBot|Pinterestbot|CMS-Checker|HeadlessChrome|Puppeteer|SeznamBot|Sogou|8LEGS|HTTrack|cherrypicker|AhrefsBot|BLEXBot|DotBot|MJ12bot|PetalBot|SemrushBot|BuiltWith|Viewer/99|Python|aiohttp|curl|Wget|libwww|Go-http-client|GeedoShopProductFinder|DuckDuckBot|node|IMJ-CompanyPage-Scraper|baidu|RootEvidence|NetAPI\sv1|Scrapy|Bingbot|SummalyBot|got|HUNT-Bot|CibraxScanner|SalesOS-CompanyVerifier|RecordedFuture|SurdotlyBot|panscient\.com|Xiaomi|Android.*Firefox|GPTBot|ClaudeBot|BardBot|LLMScraper|Firecrawl|Crawl4AI|ia_archiver|archive\.org_bot|Google|wp2shell|okhttp|Cortex-Xpanse|Mozilla\.5\.0\.compatible;\.MSIE\.10\.0;\.Windows\.NT\.6\.1;\.Trident/6\.0|facebookexternalhit|facebookexternalua|Version\.13\.0\.3\.Mobile\.15E148\.Safari\.604\.1|iPhone.*Version/13\.0\.3|Safari/604\.1|cms-scanner|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|Amazonbot|Applebot-Extended|Meta-ExternalAgent|Meta-ExternalFetcher|cohere-ai|DeepSeek|Bytespider|Diffbot|Omgilibot|Omgili|Google-Extended|Google-CloudVertex|MistralAI-User|OAI-AdsBot|YouBot|anthropic-ai|NosibleBot) [NC,OR]
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://"
[NC]RewriteRule ^.*$ - [G,L]
# 2. BLOCK SPAM WEBSITES (REFERRERS)
RewriteCond %{HTTP_REFERER} (baidu\.com|bsky\.(net|com|app)|facebook\.com|meta\.com|threads\.(com|net)|instagram\.com|google\.com|googleusercontent\.com|youtube\.com|x\.com|t\.co|twitter\.com|x\.ai|bing\.com|yahoo\.com|yandex\.com|duckduckgo\.com|microsoft\.com|amazon\.com|brave\.com|semalt\.com|buttons-for-website\.com|darodar\.com|blackhatworth\.com|ilovevitaly\.com|priceg\.com|ranksonic\.com)
[NC]RewriteRule ^.*$ - [G,L]
###############################################
# SAFE BOT & SCRAPER KILLER (FRIENDICA-COMPATIBLE)
###############################################
# Kill obvious scanners by User-Agent
RewriteCond %{HTTP_USER_AGENT} (nmap|nikto|acunetix|sqlmap|fimap|nessus|openvas|arachni|wpscan|dirbuster|fuzzer)
[NC]RewriteRule ^ - [G,L]
# Kill requests with directory traversal attempts
RewriteCond %{QUERY_STRING} (\.\./|\.\.\\|%2e%2e|%5c)
[NC]RewriteRule ^ - [G,L]
# Kill malformed absolute URLs
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://"
[NC]RewriteRule ^ - [G,L]
# Kill empty User-Agent ONLY if NOT federation
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteCond %{REQUEST_URI} !^/\.well-known/
[NC]RewriteRule ^ - [G,L] which has cut down a lot of bad traffic from getting 200, and getting hit with a 410 when it knocks on my door, I would start there, because blocking ips is fun and easy, but if they are being spoofed, they will just spoof more, and worse when they get recycled to actual valid users then you lose traffic, I have found the agent blocking is more effective, they are not willing to rewrite every script to adjust for that so for now it seems to be the fastest block, and if it is getting that hard, run it through cloudflare on the free side proxied for a while anyways to help slowdown and divert the bad traffic, it will keep your server happier, I know not everyone is a fan of that, but somtimes you have to change the route to stay on the road
@pasjrwoctx
More details regarding user-agent distribution
Retained log window: 2026-08-14 17:50–22:42 CEST
Challenge requests: 164,435
Distinct user-agent strings: 160
Share represented by the top ten: 96.79%
The almost perfectly even distribution among the first ten identities is particularly striking.
tom_s@friendica.ambag.es@friendica.ambag.es 15 hours ago