INeedMana

I haven’t hosted something big in the past but I have dabbled sometimes

The ssh knockers have always been there. But now the blocklists are either more accessible or I just have heard about them more in the meantime

Scrapers did not use to DDoS so aggressively but also have been there. Now we have the challenges (anubis, cloudfront, captchas) to mitigate their hunger (and the blocklists)

What has changed is that now one needs to remember what a piece is written in. A PHP vulnerability one day, Go another. And everytime one has to run through the list “what the stuff we’re hosting is written in”. But personally, I think this will pass. What is being discovered right now are often long-standing bugs, so the only mitigation is update. Once the vulnerabilities are coming from a recent change, sometimes it will be patched before we need the newer version of a library

Massive shout of thanks from myself to INeedMana for jumping in and spending hours fixing this with me, guiding me, and putting up with my stress and panic!

When we were hit by scrapers the other month, both load and CPU utilization were going through the roof, server was choking even on sending ssh data. So that load of all cores being requested, while uncharacteristic, didn’t automatically feel alarming. It wasn’t that bad
When I was sitting down to take a look at the cron jobs, I was not expecting to find what we’ve found. And by the time I was getting API errors on simple docker exec I was starting to run in circles

I’m very grateful that you were there, even though it was well past midnight for you. It was much more manageable to get through that in a duo

That is my understanding too. But then when I was reading the article, there was a line somewhere that in order for the exploit to be successful, the server needs to be using eval

I guess nearly everybody has installed the patch by now

I admire your belief ;)

https://php2shell.censys.com/

ATM almost 80% of scanned services are still vulnerable. Considering that **About 54% of PHP services hide their version** and are not counted as affected or patched.it doesn’t look too good

AFAIU this is not about the size of the file but making the php process that runs a file surpass the configured limit (of RAM). At which point sometimes something goes wrong with letting go of that memory and then the second vulnerability allows using that limboed memory that not only shouldn’t be used but also can contain some internal values from the first one

@povoq@slrpnk.net I think i_am_not_a_robot has a point that it’s too soon

For unrelated reason I had to wipe my blocklists on VPS for a moment. Just during a few minutes one of my containers was getting hammered with all kind of .php requests.
Even if it wasn’t used in the wild before, it seems that now bots are running in circles with those new toys

Yes, but from the page I linked

One really amusing thing in common with most of these “PoC"s is that they lean on “LFI”, or local-file-inclusion, to make the vulnerability. In non-php terms, their exploit chain requires the server to already have a severe code-level vulnerability.

It’s not rare for PHP apps to be fucked in some sense, but requiring the server to have an existing vulnerability where users could cat /etc/password is an insane pre-requisite. If you need this, this does not make this a severe bug.

For sure that this is still a bug that should be patched. But I do remember “don’t eval on user sent stuff” rule. Or is this really prevalent in PHP ecosystem?

running pods: 98

Oof, my lab is lightweight compared to yours but I’ve been thinking of putting up a grafana for mine too. Would you mind sharing how much resources it takes to run and roughly how much storage does, let’s say, a month of data take? I know that it all depends on what one puts inside but just as a reference point

Hello 👋 🙂

Last few years I’ve been rather listening to stuff like Electric Moon but for windows down I’d pick Witch - Seer

Have you heard about funkwhale software?

Might be! It’s just that having a container like that feels easier than looking after bare-metal secondary db

Maybe I’ll just pg_dump during Nextcloud maintenance window and get into BorgBackup putting the dumps on NAS? If I have to revive the db from the dead, I’ll probably just use the dump anyway, so that’s one hoop less

2nd DB would be on the NAS but its not being used by the app, so as long as it’s asynchronous writes, you should be okay

But then I have to maintain it too. And it will be using RAM that was supposed to only be for servicing RAID. On a platform where RAM can’t grow

For sure my setup will be growing and changing with time. But ATM this “prod” db lives only on laptop’s SSD

replication in the database it self

AFAIK db replication would require running another db on NAS. Which I tried to avoid

I have another cronjob that takes a backup into the first database docker init folder. Older backups end with -date so they won’t get picked up by the init script. You can do similar with just docker.

Yeah, that’s why I’m thinking to do. Have a container that watches changes to db data and rsyncs those into the NAS for safekeeping. My main goal in all of this is so db data won’t die with laptop’s SSD

Oh, that’s interesting! Sounds a bit crazy but I’m interested in what you find out :D

I wonder if in this case VMs don’t make it work better. There is another layer between the process and NFS than pure docker+volume

db running on the Pi with storage on the Pi could be faster than the db running on the laptop with storage on the Pi

I think so too, but I’m mindful of the 1GB RAM per TB rule. I’m afraid to choke the NAS and on Pi I can’t put in more RAM

keep the DB on the most powerful CPU and quickest disk

Thing is, I expect that NAS to be faster than the SSD on the laptop that runs a bunch of stuff. Even NC log file will take up some IO

But I get the point you are all making here, I’m changing course :)

it’s generally not a good plan to put anything that needs realtime access remote to the device processing it

So in your opinion putting whole nextcloud_data as NFS mount is also a bad idea? Keep NFS as external storage and/or only user data?

AIO docker

It was too blackbox to me. I had some issue and found it hard to debug. fpm+nginx is more my speed
This BorgBackup app is available only in AIO?

You can easily just try it out though

Yeah, but if that setup leads to problem in a few months, I won’t catch that. That’s why I decided to ask and judging by the other answer, I dodged a bullet here

Now I’m thinking of setting up something like this. Maybe even in another container? One volume with data bound, one with NFS and a bunch of scripts in entrypoint to rsync(?) between the two on inotifywait. There is lsyncd but it’s been unmaintained for two years now

Ah. So there is a devil in the details here. Thank you :)

back the data dir up to the NAS (or replicate)

Could you please expand on that? You mean DB replication or there is something like NFS replication?