pasjrwoctx

@pasjrwoctx@social.2ndshot.photos

This is a remote user, information on this page may be incomplete. View original ↗

⁨Comment⁩ on ⁨Another massive distributed HTTP flood is currently hitting git.friendi.ca a⁩ ⁨⁨2⁩ ⁨days⁩ ago⁩:

@utzer @utzer ok so you could block those user agents I have something like

 # Fake browser detection
RewriteCond %{HTTP_USER_AGENT} (Chrome/[0-9]{3}|Chrome/1[3-9][0-9]|Chrome/150|Firefox/1[3-9][0-9]|Safari/60[0-9]|Version/17) [NC]
RewriteCond %{HTTP_ACCEPT_LANGUAGE} ^$
RewriteRule ^ - [G,L]

RewriteCond %{THE_REQUEST} "GET\shttp" [NC]
RewriteRule ^ - [G,L]  and then # 1. BLOCK BAD BROWSER NAMES / BOT FRAMEWORKS

RewriteCond %{HTTP_USER_AGENT} (CCBot|SearchEngineBot|Pandalytics|UCBrowser|ZoneProjectBot|Embarcadero\sURI\sClient|Xenu\sLink\sSleuth|siteradar|SignalsBot|fun-cert-watch|SERankingBacklinksBot|Pinterestbot|CMS-Checker|HeadlessChrome|Puppeteer|SeznamBot|Sogou|8LEGS|HTTrack|cherrypicker|AhrefsBot|BLEXBot|DotBot|MJ12bot|PetalBot|SemrushBot|BuiltWith|Viewer/99|Python|aiohttp|curl|Wget|libwww|Go-http-client|GeedoShopProductFinder|DuckDuckBot|node|IMJ-CompanyPage-Scraper|baidu|RootEvidence|NetAPI\sv1|Scrapy|Bingbot|SummalyBot|got|HUNT-Bot|CibraxScanner|SalesOS-CompanyVerifier|RecordedFuture|SurdotlyBot|panscient\.com|Xiaomi|Android.*Firefox|GPTBot|ClaudeBot|BardBot|LLMScraper|Firecrawl|Crawl4AI|ia_archiver|archive\.org_bot|Google|wp2shell|okhttp|Cortex-Xpanse|Mozilla\.5\.0\.compatible;\.MSIE\.10\.0;\.Windows\.NT\.6\.1;\.Trident/6\.0|facebookexternalhit|facebookexternalua|Version\.13\.0\.3\.Mobile\.15E148\.Safari\.604\.1|iPhone.*Version/13\.0\.3|Safari/604\.1|cms-scanner|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|Amazonbot|Applebot-Extended|Meta-ExternalAgent|Meta-ExternalFetcher|cohere-ai|DeepSeek|Bytespider|Diffbot|Omgilibot|Omgili|Google-Extended|Google-CloudVertex|MistralAI-User|OAI-AdsBot|YouBot|anthropic-ai|NosibleBot) [NC,OR]
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://" [NC]
RewriteRule ^.*$ - [G,L]

# 2. BLOCK SPAM WEBSITES (REFERRERS)
RewriteCond %{HTTP_REFERER} (baidu\.com|bsky\.(net|com|app)|facebook\.com|meta\.com|threads\.(com|net)|instagram\.com|google\.com|googleusercontent\.com|youtube\.com|x\.com|t\.co|twitter\.com|x\.ai|bing\.com|yahoo\.com|yandex\.com|duckduckgo\.com|microsoft\.com|amazon\.com|brave\.com|semalt\.com|buttons-for-website\.com|darodar\.com|blackhatworth\.com|ilovevitaly\.com|priceg\.com|ranksonic\.com) [NC]
RewriteRule ^.*$ - [G,L]

###############################################
# SAFE BOT & SCRAPER KILLER (FRIENDICA-COMPATIBLE)
###############################################

# Kill obvious scanners by User-Agent
RewriteCond %{HTTP_USER_AGENT} (nmap|nikto|acunetix|sqlmap|fimap|nessus|openvas|arachni|wpscan|dirbuster|fuzzer) [NC]
RewriteRule ^ - [G,L]

# Kill requests with directory traversal attempts
RewriteCond %{QUERY_STRING} (\.\./|\.\.\\|%2e%2e|%5c) [NC]
RewriteRule ^ - [G,L]

# Kill malformed absolute URLs
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://" [NC]
RewriteRule ^ - [G,L]

# Kill empty User-Agent ONLY if NOT federation
RewriteCond %{HTTP_USER_AGENT} ^$ 
RewriteCond %{REQUEST_URI} !^/\.well-known/ [NC]
RewriteRule ^ - [G,L]

which has cut down a lot of bad traffic from getting 200, and getting hit with a 410 when it knocks on my door, I would start there, because blocking ips is fun and easy, but if they are being spoofed, they will just spoof more, and worse when they get recycled to actual valid users then you lose traffic, I have found the agent blocking is more effective, they are not willing to rewrite every script to adjust for that so for now it seems to be the fastest block, and if it is getting that hard, run it through cloudflare on the free side proxied for a while anyways to help slowdown and divert the bad traffic, it will keep your server happier, I know not everyone is a fan of that, but somtimes you have to change the route to stay on the road

@ utzer @ utzer ok so you could block those user agents I have
Submitted ⁨⁨2⁩ ⁨days⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨Another massive distributed HTTP flood is currently hitting git.friendi.ca a⁩ ⁨⁨2⁩ ⁨days⁩ ago⁩:
@utzer ok so you could block those user agents I have something like # Fake browser detection
RewriteCond %{HTTP_USER_AGENT} (Chrome/[0-9]{3}|Chrome/1[3-9][0-9]|Chrome/150|Firefox/1[3-9][0-9]|Safari/60[0-9]|Version/17)
[NC]RewriteCond %{HTTP_ACCEPT_LANGUAGE} ^$
RewriteRule ^ - [G,L]
RewriteCond %{THE_REQUEST} "GET\shttp"
[NC]RewriteRule ^ - [G,L] and then # 1. BLOCK BAD BROWSER NAMES / BOT FRAMEWORKS
RewriteCond %{HTTP_USER_AGENT} (CCBot|SearchEngineBot|Pandalytics|UCBrowser|ZoneProjectBot|Embarcadero\sURI\sClient|Xenu\sLink\sSleuth|siteradar|SignalsBot|fun-cert-watch|SERankingBacklinksBot|Pinterestbot|CMS-Checker|HeadlessChrome|Puppeteer|SeznamBot|Sogou|8LEGS|HTTrack|cherrypicker|AhrefsBot|BLEXBot|DotBot|MJ12bot|PetalBot|SemrushBot|BuiltWith|Viewer/99|Python|aiohttp|curl|Wget|libwww|Go-http-client|GeedoShopProductFinder|DuckDuckBot|node|IMJ-CompanyPage-Scraper|baidu|RootEvidence|NetAPI\sv1|Scrapy|Bingbot|SummalyBot|got|HUNT-Bot|CibraxScanner|SalesOS-CompanyVerifier|RecordedFuture|SurdotlyBot|panscient\.com|Xiaomi|Android.*Firefox|GPTBot|ClaudeBot|BardBot|LLMScraper|Firecrawl|Crawl4AI|ia_archiver|archive\.org_bot|Google|wp2shell|okhttp|Cortex-Xpanse|Mozilla\.5\.0\.compatible;\.MSIE\.10\.0;\.Windows\.NT\.6\.1;\.Trident/6\.0|facebookexternalhit|facebookexternalua|Version\.13\.0\.3\.Mobile\.15E148\.Safari\.604\.1|iPhone.*Version/13\.0\.3|Safari/604\.1|cms-scanner|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|Amazonbot|Applebot-Extended|Meta-ExternalAgent|Meta-ExternalFetcher|cohere-ai|DeepSeek|Bytespider|Diffbot|Omgilibot|Omgili|Google-Extended|Google-CloudVertex|MistralAI-User|OAI-AdsBot|YouBot|anthropic-ai|NosibleBot) [NC,OR]
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://"
[NC]RewriteRule ^.*$ - [G,L]
# 2. BLOCK SPAM WEBSITES (REFERRERS)
RewriteCond %{HTTP_REFERER} (baidu\.com|bsky\.(net|com|app)|facebook\.com|meta\.com|threads\.(com|net)|instagram\.com|google\.com|googleusercontent\.com|youtube\.com|x\.com|t\.co|twitter\.com|x\.ai|bing\.com|yahoo\.com|yandex\.com|duckduckgo\.com|microsoft\.com|amazon\.com|brave\.com|semalt\.com|buttons-for-website\.com|darodar\.com|blackhatworth\.com|ilovevitaly\.com|priceg\.com|ranksonic\.com)
[NC]RewriteRule ^.*$ - [G,L]
###############################################
# SAFE BOT & SCRAPER KILLER (FRIENDICA-COMPATIBLE)
###############################################
# Kill obvious scanners by User-Agent
RewriteCond %{HTTP_USER_AGENT} (nmap|nikto|acunetix|sqlmap|fimap|nessus|openvas|arachni|wpscan|dirbuster|fuzzer)
[NC]RewriteRule ^ - [G,L]
# Kill requests with directory traversal attempts
RewriteCond %{QUERY_STRING} (\.\./|\.\.\\|%2e%2e|%5c)
[NC]RewriteRule ^ - [G,L]
# Kill malformed absolute URLs
RewriteCond %{THE_REQUEST} "^[A-Z]{3,9}\s+https?://"
[NC]RewriteRule ^ - [G,L]
# Kill empty User-Agent ONLY if NOT federation
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteCond %{REQUEST_URI} !^/\.well-known/
[NC]RewriteRule ^ - [G,L] which has cut down a lot of bad traffic from getting 200, and getting hit with a 410 when it knocks on my door, I would start there, because blocking ips is fun and easy, but if they are being spoofed, they will just spoof more, and worse when they get recycled to actual valid users then you lose traffic, I have found the agent blocking is more effective, they are not willing to rewrite every script to adjust for that so for now it seems to be the fastest block, and if it is getting that hard, run it through cloudflare on the free side proxied for a while anyways to help slowdown and divert the bad traffic, it will keep your server happier, I know not everyone is a fan of that, but somtimes you have to change the route to stay on the road
@ utzer ok so you could block those user agents I have something like # F
Submitted ⁨⁨2⁩ ⁨days⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨Another massive distributed HTTP flood is currently hitting git.friendi.ca a⁩ ⁨⁨2⁩ ⁨days⁩ ago⁩:
@utzer what is the user agents, looking at your access server logs, you should be able to see if even you had 1million different ip hitting at the same time, if all the user agents are the same, then you have trace back all million ip between whois and dig, to figure out if they are tied together, if all same user agent and time each ip hit and pulls same amount of data, that leans more to coordinated DDoS, if differing user agents, differing hit times and data captures, then massive scraping
did I make any sense
@ utzer what is the user agents, looking at your access server logs, you
Submitted ⁨⁨2⁩ ⁨days⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨Does anyone have a dummies guide to setting up # Cloudflare for # frie⁩ ⁨⁨6⁩ ⁨days⁩ ago⁩:
So I think I got #Cloudflare all setup now, and supposedly this instance is running through it, but I have no idea if anything at all is really happening;
Does anyone have a dummies guide to setting up # Cloudflare for # frie
Submitted ⁨⁨6⁩ ⁨days⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨1⁩ ⁨comment⁩
Ok so video uploads to # friendica kind of suck, I get it, it is not exacyour.instance.domain ↗
Submitted ⁨⁨5⁩ ⁨weeks⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
Ok so video uploads to # friendica kind of suck, I get it, it is not exacyour.instance.domain ↗
Submitted ⁨⁨5⁩ ⁨weeks⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨So I kind of set up another level of security well not really more of a gate to ⁩ ⁨⁨1⁩ ⁨month⁩ ago⁩:
@webfan the other reason I elected to not have a server side stored is one privacy of actual humans that visit my site directly, and I just rather not be collecting useless data on a fairly restricted shared hosting environment, I am sure with a little time and a bit more effort on my part I could build fort knox around my instance, but the whole point of this was to cut down on the constant abusive bots and scrappers, and at this time it appears to have worked or is working as is;
@ Melanie Wehowski the other reason I elected to not have a server side s
Submitted ⁨⁨1⁩ ⁨month⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨So I kind of set up another level of security well not really more of a gate to ⁩ ⁨⁨1⁩ ⁨month⁩ ago⁩:
@webfan also if you wanted to go that far as adding /inbox following the logic you should probably also add /nodeinfo as well;
@ Melanie Wehowski also if you wanted to go that far as adding /inbox fol
Submitted ⁨⁨1⁩ ⁨month⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨So I kind of set up another level of security well not really more of a gate to ⁩ ⁨⁨1⁩ ⁨month⁩ ago⁩:
@webfan cookie could be faked but so can headers, and the way I have it now the cookie is only good for the session, bots and scrapers start a new session every time they probe a domain so no need to store, each time they troll its a new session, putting inbox in the check I guess could be useful to gate it, but when I sat down to do this and dug through all my access logs I did not have even one call for the inbox to be worried about it, now if I get bots or scrapers that start calling for my inbox I will gate it, but my understanding of how friendica and activitypub are setup the inbox should not be callable by bots and scrapers, I maybe wrong on that, I have had this instance up an running a year, and it has be en a lot of learning and trial and error so many errors, but so far this gate has really made my access logs go empty, which for me is good, im still fully federated but the datahogs have slowed or stopped trolling me
@ Melanie Wehowski cookie could be faked but so can headers, and the way
Submitted ⁨⁨1⁩ ⁨month⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨So I kind of set up another level of security well not really more of a gate to ⁩ ⁨⁨1⁩ ⁨month⁩ ago⁩:
@webfan the only reason to exclude the /inbox route is if you are finding a lot of bots and scrapers accessing your inbox before they scrape your profile, in theory friendicas rules and activitypubs rules should already have your inbox blocked from that activity, I created this because even with a very robust .htaccess file I was getting a lot of bots and scrapers accessing "GET" and "Profile" in my access logs, most where defeating my robots.txt and breaking the rules in .htaccess, this at the moment has killed them, first 2 days they all got 403's and then they just stopped trolling my instance at all; I think it could be customized to your individual needs, I made it shareable, I wont maintain it, it works for me and if by chance it helps anyone well all the better, by all means adapt and modify it to your instance's needs and if it works share it with the community, security and privacy in numbers I say;
@ Melanie Wehowski the only reason to exclude the /inbox route is if you
Submitted ⁨⁨1⁩ ⁨month⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
⁨Comment⁩ on ⁨So I kind of set up another level of security well not really more of a gate to ⁩ ⁨⁨1⁩ ⁨month⁩ ago⁩:
@csolisr I have no idea about making it an addon, personally I think that defeats the purpose, as this really acts like the first line of defense after .htaccess rules, before anything else even happens, at least that is how I saw it working;
@ Carlos Solís I have no idea about making it an addon, personally I thin
Submitted ⁨⁨1⁩ ⁨month⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨0⁩ ⁨comments⁩
So I kind of set up another level of security well not really more of a gate to
Submitted ⁨⁨2⁩ ⁨months⁩ ago⁩ to ⁨helpers@forum.friendi.ca⁩ | ⁨12⁩ ⁨comments⁩