How I accidentally slowed down my nextcloud instance for months

Submitted ⁨⁨9⁩ ⁨months⁩ ago⁩ by ⁨Moonrise2473@feddit.it⁩ to ⁨selfhosted@lemmy.world⁩

I am running this docker image: github.com/nextcloud/docker with a cloudflare tunnel, meaning the webserver would see all the traffic coming from a single ip in 172.16.0.0/12 .

The documentation says:

The apache image will replace the remote addr (IP address visible to Nextcloud) with the IP address from X-Real-IP if the request is coming from a proxy in 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16 by default

So I thought that this is a not a problem, as other docker images can also automagically figure out the real IP address from traffic coming from cloudflare tunnels.

In the beginning it worked fine, then it was SLOW. Like 2 full minutes to load new feeds on news, waiting ages to complete a sync, and so on. I rebooted the server on those instances, and then it worked fine for a day.

So because at the time i was running it on unraid, i blamed the lag on that OS + my weird array of HDDs with decades of usage on them. Migrated to debian on a nvme array and… same lag!

Wasted hours trying to use caddy+fpm instead of apache and it’s the same, worked fine for a day, then it was slow again.

Then I wondered: what if the program is “smart” and throttles it by itself without any warning to the admin if it thinks that an ip address is sending too many requests?

Modified the docker compose like this:

  nextcloud:
    image: nextcloud

became

  nextcloud:
    build: .

and I created a Dockerfile with

FROM nextcloud
RUN apt update -y && apt upgrade -y
RUN apt install -y libbz2-dev
RUN docker-php-ext-install bz2
RUN a2enmod rewrite remoteip
COPY remoteip.conf /etc/apache2/conf-enabled/remoteip.conf

with this as the content of remoteip.conf

RemoteIPHeader CF-Connecting-IP
RemoteIPTrustedProxy 10.0.0.0/8
RemoteIPTrustedProxy 172.16.0.0/12
RemoteIPTrustedProxy 192.168.0.0/16
RemoteIPTrustedProxy 173.245.48.0/20
RemoteIPTrustedProxy 103.21.244.0/22
RemoteIPTrustedProxy 103.22.200.0/22
RemoteIPTrustedProxy 103.31.4.0/22
RemoteIPTrustedProxy 141.101.64.0/18
RemoteIPTrustedProxy 108.162.192.0/18
RemoteIPTrustedProxy 190.93.240.0/20
RemoteIPTrustedProxy 188.114.96.0/20
RemoteIPTrustedProxy 197.234.240.0/22
RemoteIPTrustedProxy 198.41.128.0/17
RemoteIPTrustedProxy 162.158.0.0/15
RemoteIPTrustedProxy 104.16.0.0/12
RemoteIPTrustedProxy 172.64.0.0/13
RemoteIPTrustedProxy 131.0.72.0/22
RemoteIPTrustedProxy 2400:cb00::/32
RemoteIPTrustedProxy 2606:4700::/32
RemoteIPTrustedProxy 2803:f800::/32
RemoteIPTrustedProxy 2405:b500::/32
RemoteIPTrustedProxy 2405:8100::/32
RemoteIPTrustedProxy 2a06:98c0::/29
RemoteIPTrustedProxy 2c0f:f248::/32

and now because nextcloud is seeing all the different ip addresses it doesn’t throttle the connections anymore!

source

Comments

Sort:hotnew top

rufus@discuss.tchncs.de ⁨9⁩ ⁨months⁩ ago
Why do so many people tunnel their data through cloudflare anyways? No port forwarding possible? Or afraid of DDoS attacks? Or am I missing something?

source
- chiisana@lemmy.chiisana.net ⁨9⁩ ⁨months⁩ ago
  Security.
  
  Cloudflare handles a very large amount of traffic and sees many different types of attacks (thinks CSRF, injections, etc.). It is unlikely that you or me will be individually targeted, but drive-bys are a thing, and thanks to the amount of traffic they monitor, the WAF will more likely block out anything and patch before I’m able to update my apps on 0 days.
  
  Also, while WAF is a paid feature, other free features, such as free DDOS attack protection, help prevent other attacks.
  
  It’s a trade off, sure; they’re technically MITM’ing your traffic, but frankly, I don’t care. Much like no one cares to target/attack me individually, they aren’t going to look at my content individually.
  
  Additionally, it also makes accessing things much easier. Also, it is much more likely I’d find a SME using Cloudflare than some janky custom self hosted tunnel setup. So from a using homelab as a learning for professional experience point of view, it is much more applicable as well.
  
  source
  - lemmyvore@feddit.nl ⁨9⁩ ⁨months⁩ ago
    If they don’t care to attack you why would they DDoS you. 😄
    
    The things CF fans make up about “security” are hilarious.
    
    If you ever got hit with a DDoS while on the free tier they’d just disconnect you.
    
    source
    -> View More Comments
  - rufus@discuss.tchncs.de ⁨9⁩ ⁨months⁩ ago
    Thx for explaining. I’m not sure if I’m willing to do the same trade-offs. Supposedly their WAF is very good and quite some people use it. Probably for a good reason… It just comes at a hefty price. I’m doing selfhosting to emancipate myself, stay independent and in control. I’m not sure if becoming dependant on a single large company and terminating my encryption on their servers that do arbitrary magic and whatever with my packets is something that aligns with my ethics.
    
    source
    -> View More Comments
  - possiblylinux127@lemmy.zip ⁨9⁩ ⁨months⁩ ago
    I just use a VPS with caching and basic https stripping protection
    
    source
  - atzanteol@sh.itjust.works ⁨9⁩ ⁨months⁩ ago
    Nobody is going to go through the effort to ddos a personal site. 😂
    
    source
    -> View More Comments
- lemmyvore@feddit.nl ⁨9⁩ ⁨months⁩ ago
  They think the free CF tier offers DDoS protection, which (a) will never happen to their server and (b) if it ever happened would consist of CF disconnecting their tunnel and black-holing their IP and domain until it blows over.
  
  They also think CDN helps when your services are behind authentication.
  
  Some of them just find it convenient that CF is registrar, DNS provider and sets up reverse proxy for them so they never stop to think too much about it.
  
  source
  - rufus@discuss.tchncs.de ⁨9⁩ ⁨months⁩ ago
    Thanks.
    
    source
  - dan@upvote.au ⁨9⁩ ⁨months⁩ ago
    
    Some of them just find it convenient that CF is registrar, DNS provider and sets up reverse proxy
    
    You should never put all your eggs in one basket. Using one company for all three of these essentially gives them full control of your domain.
    
    It’s a best practice to use separate companies for registrar and website/proxy. If there’s ever some sort of dispute about the contents of the site, you can change the DNS to point to a different host. That’s not always possible when the same company handles both.
    
    source
- Moonrise2473@feddit.it ⁨9⁩ ⁨months⁩ ago
  Simple reason: at home I don’t have a static IPv4 address and I can’t do port forwarding
  
  source
  - caleb@lemmy.moorenet.casa ⁨9⁩ ⁨months⁩ ago
    What about ddns?
    
    source
  - rufus@discuss.tchncs.de ⁨9⁩ ⁨months⁩ ago
    Thx, that is a good reason to do it. I’m eventually going to lose my static IPv4 address, too. But I’m preparing to move some to a VPS instead and in the process set up the firewall and the reverse proxy to the Nextcloud and so on on my homeserver there.
    
    source
    -> View More Comments
  - dan@upvote.au ⁨9⁩ ⁨months⁩ ago
    Get a $15/year VPS and run your own tunnel using Wireguard.
    
    source
- Gutless2615@ttrpg.network ⁨9⁩ ⁨months⁩ ago
  Is there a better way to expose my services when being a cgnat?
  
  source
  - rufus@discuss.tchncs.de ⁨9⁩ ⁨months⁩ ago
    Cloudflare, Pagekite, a cheap VPS with a reverse proxy. Maybe IPv6-only access if your CGNat does that, ngrok, a VPN… There are quite some tools and services available. And which one is right for you might depend on the exact situation and what you’re hosting. I’m not an expert on this. I have an internet connection without a NAT, and additionally a really tiny VPS with a mailserver, a small website and wireguard. I just use that to tunnel through NAT if i need to. But that means I haven’t compared all the other services since I don’t need them (yet.)
    
    source
- EncryptKeeper@lemmy.world ⁨9⁩ ⁨months⁩ ago
  Because it makes them “feel” more secure.
  
  source
Netrunner@programming.dev ⁨9⁩ ⁨months⁩ ago
I just don’t see the point of using cloudflared. Its easy to use but it just gives all your data to cloudflared in return for very little.

source
- Moonrise2473@feddit.it ⁨9⁩ ⁨months⁩ ago
  You can easily set SSL with a self signed certificate, they get nothing
  
  source
  - Netrunner@programming.dev ⁨9⁩ ⁨months⁩ ago
    Afaik, they decrypt and recrypt all traffic.
    
    source
    -> View More Comments
doeknius_gloek@discuss.tchncs.de ⁨9⁩ ⁨months⁩ ago
Interesting - I didn’t bother to set the X-Real-IP headers until now and this might speed up my instance too. Thanks!

Then I wondered: what if the program is “smart” and throttles it by itself without any warning to the admin if it thinks that an ip address is sending too many requests?

The word you’re looking for is “Rate Limit(ing)” and according to the documentation you could also disable it completely.

But I guess the cleanest and most secure solution would be to just set the headers on the reverse proxy.

source
teawrecks@sopuli.xyz ⁨9⁩ ⁨months⁩ ago
Good job debugging it. Where’d you get that list of IPs?

source
- Moonrise2473@feddit.it ⁨9⁩ ⁨months⁩ ago
  It’s the list of IPs that belong to cloudflare.
  
  I think that because I’m using tunnels it’s not necessary to have all of them, just the docker ip address space
  
  source
Decronym@lemmy.decronym.xyz ⁨9⁩ ⁨months⁩ ago
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters

DNS Domain Name Service/System

IP Internet Protocol

VPS Virtual Private Server (opposed to shared hosting)

[Thread #484 for this sub, first seen 4th Feb 2024, 22:45] [FAQ] [Full list] [Contact] [Source code]

source
JVT038@feddit.nl ⁨9⁩ ⁨months⁩ ago
Does this also apply to linuxserver/nextcloud image? Because that’s what I’m using.

source
- zoontechnicon@feddit.de ⁨9⁩ ⁨months⁩ ago
  yes
  
  source

Fewer Letters	More Letters
DNS	Domain Name Service/System
IP	Internet Protocol
VPS	Virtual Private Server (opposed to shared hosting)