is there anyone who can help to figure out if this is a huge scraping or a DDoS?
Comment on Another massive distributed HTTP flood is currently hitting git.friendi.ca a
utzer@f.utzer.de@f.utzer.de 3 weeks ago Update: The flood is still ongoing and intensified again today.
Between approximately 17:50 and 22:03 CEST, Anubis recorded at least 141,090 challenges from 136,393 different IP addresses, peaking at roughly 35,800 challenges per hour.
Most of the current traffic is still targeting the friendica-addons pull-request list. Our temporary rate limit is successfully keeping this traffic away from the backend: the rest of git.friendi.ca remains fast and accessible, normal Git access works, and server load is currently back to normal.
The affected pull-request page may continue to return “Too Many Requests” while the flood persists.
- utzer@f.utzer.de@f.utzer.de 3 weeks ago
pasjrwoctx@social.2ndshot.photos 3 weeks ago
@utzer what is the user agents, looking at your access server logs, you should be able to see if even you had 1million different ip hitting at the same time, if all the user agents are the same, then you have trace back all million ip between whois and dig, to figure out if they are tied together, if all same user agent and time each ip hit and pulls same amount of data, that leans more to coordinated DDoS, if differing user agents, differing hit times and data captures, then massive scraping
did I make any sense- utzer@f.utzer.de@f.utzer.de 3 weeks ago
@pasjrwoctx Thanks, I checked the user agents. The pattern looks highly coordinated.
Out of 164,435 Anubis challenge requests, there were only 160 distinct user-agent strings. The top ten account for 96.79% of all requests and were used almost perfectly evenly:
- Firefox 120 / Windows — 16,066
- Chrome 118 / Windows — 16,052
- Firefox 121 / macOS — 15,949
- Edge 119 / Windows — 15,917
- Chrome 119 / Windows — 15,874
- Chrome 119 / macOS — 15,872
- Edge 120 / Windows — 15,863
- Chrome 120 / macOS — 15,853
- Firefox 121 / Windows — 15,812
- Chrome 120 / Windows — 15,761
These are old browser versions from around 2023. Combined with almost one IP per request, this looks like a single coordinated tool rotating through a fixed list of spoofed browser identities and a very large residential/proxy network. It does not prove whether the objective is scraping or disruption, but it is clearly not ordinary independent crawling.
pasjrwoctx@social.2ndshot.photos 3 weeks ago
@utzer ok so you could block those user agents I have something like # Fake browser detection
RewriteCond %{HTTP_USER_AGENT} (Chrome/[0-9]{3}|Chrome/1[3-9][0-9]|Chrome/150|Firefox/1[3-9][0-9]|Safari/60[0-9]|Version/17)
[NC]RewriteCond %{HTTP_ACCEPT_LANGUAGE} ^$
RewriteRule ^ - [G,L]RewriteCond %{THE_REQUEST} "GET\shttp"
[NC]RewriteRule ^ - [G,L] and then # 1. BLOCK BAD BROWSER NAMES / BOT FRAMEWORKSRewriteCond %{HTTP_USER_AGENT} (CCBot|SearchEngineBot|Pandalytics|UCBrowser|ZoneProjectBot|Embarcadero\sURI\sClient|Xenu\sLink\sSleuth|siteradar|SignalsBot|fun-cert-watch|SERankingBacklinksBot|Pinterestbot|CMS-Checker|HeadlessChrome|Puppeteer|SeznamBot|Sogou|8LEGS|HTTrack|cherrypicker|AhrefsBot|BLEXBot|DotBot|MJ12bot|PetalBot|SemrushBot|BuiltWith|Viewer/99|Python|aiohttp|curl|Wget|libwww|Go-http-client|GeedoShopProductFinder|DuckDuckBot|node|IMJ-CompanyPage-Scraper|baidu|RootEvidence|NetAPI\sv1|Scrapy|Bingbot|SummalyBot|got|HUNT-Bot|CibraxScanner|SalesOS-CompanyVerifier|RecordedFuture|SurdotlyBot|panscient\.com|Xiaomi|Android.*Firefox|GPTBot|ClaudeBot|BardBot|LLMScraper|Firecrawl|Crawl4AI|ia_archiver|archive\.org_bot|Google|wp2shell|okhttp|Cortex-Xpanse|Mozilla\.5\.0\.compatible;\.MSIE\.10\.0;\.Windows\.NT\.6\.1;\.Trident/6\.0|facebookexternalhit|facebookexternalua|Version\.13\.0\.3\.Mobile\.15E148\.Safari\.604\.1|iPhone.*Version/13\.0\.3|Safari/604\.1|cms-scanner|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|Amazonbot|Applebot-Extended|Meta-ExternalAgent|Meta-ExternalFetcher|cohere-ai|DeepSeek|Bytespider|Diffbot|Omgilibot|Omgili|Google-Extended|Google-CloudVertex|MistralAI-User|OAI-AdsBot|YouBot|anthropic-ai|NosibleBot) [NC,OR]
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://"
[NC]RewriteRule ^.*$ - [G,L]# 2. BLOCK SPAM WEBSITES (REFERRERS)
RewriteCond %{HTTP_REFERER} (baidu\.com|bsky\.(net|com|app)|facebook\.com|meta\.com|threads\.(com|net)|instagram\.com|google\.com|googleusercontent\.com|youtube\.com|x\.com|t\.co|twitter\.com|x\.ai|bing\.com|yahoo\.com|yandex\.com|duckduckgo\.com|microsoft\.com|amazon\.com|brave\.com|semalt\.com|buttons-for-website\.com|darodar\.com|blackhatworth\.com|ilovevitaly\.com|priceg\.com|ranksonic\.com)
[NC]RewriteRule ^.*$ - [G,L]###############################################
# SAFE BOT & SCRAPER KILLER (FRIENDICA-COMPATIBLE)
################################################ Kill obvious scanners by User-Agent
RewriteCond %{HTTP_USER_AGENT} (nmap|nikto|acunetix|sqlmap|fimap|nessus|openvas|arachni|wpscan|dirbuster|fuzzer)
[NC]RewriteRule ^ - [G,L]# Kill requests with directory traversal attempts
RewriteCond %{QUERY_STRING} (\.\./|\.\.\\|%2e%2e|%5c)
[NC]RewriteRule ^ - [G,L]# Kill malformed absolute URLs
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://"
[NC]RewriteRule ^ - [G,L]# Kill empty User-Agent ONLY if NOT federation
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteCond %{REQUEST_URI} !^/\.well-known/
[NC]RewriteRule ^ - [G,L] which has cut down a lot of bad traffic from getting 200, and getting hit with a 410 when it knocks on my door, I would start there, because blocking ips is fun and easy, but if they are being spoofed, they will just spoof more, and worse when they get recycled to actual valid users then you lose traffic, I have found the agent blocking is more effective, they are not willing to rewrite every script to adjust for that so for now it seems to be the fastest block, and if it is getting that hard, run it through cloudflare on the free side proxied for a while anyways to help slowdown and divert the bad traffic, it will keep your server happier, I know not everyone is a fan of that, but somtimes you have to change the route to stay on the road- utzer@f.utzer.de@f.utzer.de 3 weeks ago
More details regarding user-agent distribution
Retained log window: 2026-08-14 17:50–22:42 CEST
Challenge requests: 164,435
Distinct user-agent strings: 160
Share represented by the top ten: 96.79%The almost perfectly even distribution among the first ten identities is particularly striking.
COUNT USER AGENT ------ ------------------------------------------------------------ 16106 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 16097 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0 15993 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0 15959 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0 15924 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 15914 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 15899 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 15887 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0 15855 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0 15800 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2871 Mozilla/5.0 (compatible; crawler) 709 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.0 Safari/537.36 413 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 86 Chrome/135.0.0.0 — Windows 40 Chrome/116.0.0.0 — Windows 40 Chrome/107.0.0.0 — Windows 39 Chrome/148.0.0.0 — macOS 34 Chrome/150.0.0.0 — Windows 33 Chrome/131.0.0.0 — Windows 32 Chrome/133.0.0.0 — Windows 25 Edge/135.0.0.0 — Windows 24 Chrome/146.0.0.0 — macOS 23 Chrome/145.0.0.0 — Windows 23 Chrome/110.0.0.0 — Windows 22 Chrome/126.0.0.0 — Linux 22 Chrome/123.0.0.0 — macOS 21 Chrome/146.0.0.0 — Windows 21 Chrome/109.0.0.0 — Windows 21 Chrome/136.0.0.0 — macOS 20 Chrome/99.0.4844.51 — Windows 20 Chrome/149.0.0.0 — Windows 20 Chrome/145.0.0.0 — macOS 19 Chrome/133.0.0.0 — macOS 18 Chrome/105.0.0.0 — Windows 17 Chrome/134.0.0.0 — Windows 17 Chrome/111.0.0.0 — Windows 17 Safari/15.3 — macOS 17 Chrome/131.0.0.0 — macOS 16 Edge/99.0.1150.30 — Windows 16 Chrome/104.0.5112.81 — Windows 16 Firefox/135.0 — macOS 16 Safari/18.0 — macOS 16 Safari/15.5 — macOS 15 Chrome/100.0.4896.75 — Windows 15 Chrome/149.0.0.0 — macOS 14 Firefox/137.0 — Windows 14 Chrome/147.0.0.0 — Windows 14 Chrome/124.0.0.0 — Windows 14 Chrome/117.0.0.0 — Windows 14 Chrome/112.0.0.0 — Windows 14 Chrome/108.0.0.0 — Windows 14 Chrome/101.0.4951.67 — Windows 14 Safari/26.0 — macOS 14 Safari/17.0 — macOS 14 Chrome/124.0.0.0 — macOS 13 Chrome/104.0.0.0 — Windows 13 Safari/18.4 — macOS 13 Chrome/150.0.0.0 — macOS 12 Firefox/133.0 — macOS 12 Chrome/135.0.0.0 — macOS 11 Chrome/148.0.0.0 — Windows 11 Chrome/103.0.0.0 — Windows 10 Edge/101.0.1210.47 — Windows 10 Safari/18.3.1 — macOS 10 Chrome/134.0.0.0 — macOS 9 Chrome/106.0.5249.119 — Windows 9 Chrome/147.0.0.0 — macOS 8 Chrome/142.0.0.0 — Windows 7 Firefox/137.0 — Ubuntu Linux 7 Firefox/125.0 — Linux 7 Chrome/132.0.0.0 — Linux 7 Chrome/151.0.0.0 — Windows 7 Chrome/130.0.0.0 with CCleaner — Windows 7 Chrome/139.0.0.0 with Safari WebKit string — macOS 6 Chrome/137.0.0.0 — Linux 6 Chrome/114.0.0.0 — Linux 6 Firefox/153.0 — Windows 6 Edge/151.0.0.0 — Windows 5 Chrome/131.0.0.0 — Linux 5 Chrome/134.0.0.0 — ChromeOS 5 Chrome/106.0.0.0 — Windows 5 Chrome/128.0.0.0 — macOS 5 Chrome/151.0.0.0 — Android 4 Firefox/136.0 — Ubuntu Linux 4 Chrome/129.0.0.0 — Linux 4 Firefox/140.0 — Windows 4 Firefox/135.0 — Windows 4 Chrome/138.0.0.0 — Windows 4 Edge/136.0.0.0 — Windows 4 Chrome/129.0.0.0 — Windows 4 Safari/17.6 — macOS 4 Safari/17.5 — macOS 4 Chrome/147.0.0.0 — Android 3 Chrome/136.0.0.0 — Linux 3 Chrome/130.0.0.0 — Linux 3 Chrome/124.0.0.0 — Linux 3 Chrome/133.0.0.0 — ChromeOS 3 Obsidian/1.8.10, Electron/34.2.0 — Windows 3 Chrome/58.0.3029.110 — Windows 3 Edge/138.0.0.0 — Windows
rainer@friendica.sokoll.com 3 weeks ago
- utzer@f.utzer.de@f.utzer.de 2 weeks ago
Further update: I have replaced the temporary path-specific rate limit with an exact User-Agent filter.
The ten evenly rotated browser identities responsible for 96.79% of the Anubis challenges, plus the explicit `Mozilla/5.0 (compatible; crawler)` identity, are now rejected by Nginx before reaching Anubis or Forgejo.
The pull-request and issue pages are available normally again, regular web and Git access remain unaffected, and backend load is close to zero.
The crawler has since introduced a new identity:
Android 6 / Nexus 5 / Chrome 65This identity produced 527 requests from 526 different IPv4 addresses within one hour, confirming the rotating proxy-network pattern. However, the overall volume reaching Anubis has dropped from roughly 35,800 challenges per hour at the peak to about 1,000 per hour, so the exact filtering is currently working very well.
This remains a temporary mitigation because User-Agent strings can be changed at any time.