On the 24th October (heading into the 25th) Piefed.zip underwent scheduled maintenance to update to the latest version, including a full server update. At the time, it all appeared to work fine and Piefed.zip came back up as expected. On the 26th, Piefed started reporting cron job errors (A cron job is an automated, time-based task scheduled to run on the server): Some cron tasks have not been run recently: daily_maintenance (expected every None, last run at 2026-10-24 03:18 UTC), send_missed_notifs (expected every None, last run at 2026-10-26 12:01 UTC), process_email_bounces (expected every None, last run at 2026-10-26 12:01 UTC), clean_up_old_activities (expected every None, last run at 2026-10-26 12:01 UTC), send_queue (expected every None, last run at 2026-10-26 12:05 UTC) At the same time, checking the server performance we saw the CPU was being maxed out: Image (please excuse the crappy mobile screenshot) This was obviously unexpected. At this point my suspicion was a cron job had stuck in some unexpected way and caused the cpu to just shit the bed, but checking docker showed another error when attempting to initialise container security filters. Specifically, libseccomp reported an effective API level of 1 despite the installed library meeting the version requirements for the requested functionality. This behaviour was inconsistent with the expected capabilities of the server. Further investigation here led to the discovery of unauthorised processes and persistence mechanisms. (Although interference from malicious software is a possible explanation, the investigation did not establish a direct causal relationship between the malware and the seccomp failures.) At this point I took the piefed and infrastructure stacks we use offline, and started trying to figure out what was going on. Based on the hidden artifacts and persistence mechanisms, I was able to tie this to perfctl, which is a stealthy, persistent Linux malware family designed primarily for cryptomining (XMRIG Monero miners). This then led to looking for the attack vector. It is known for targetting exposed portainer agents (usually port 9001) and sure enough we had an exposed portainer agent. We (over 12 months ago) used portainer to manage our servers without needing to ssh in to them. It should have been removed from all our servers a long time ago, however on the piefed server the old portainer agent container was stopped but not deleted. At some point during the maintenance this has been restarted by mistake, and it looks like it was brought up in its default state. Regardless of this, the fact this could ever happen is totally my fault and all blame lies at my feet for not tidying up properly. So now we knew the attack vector and that the attack took place sometime on the 24th leading into the 25th (based on the graphs first showing the cpu maxing out, it was the early hours of the 25th). At this point both INeedMana and myself were able to start a system restore process. Luckily Hetzner provides a really really handy backup system, and we reimaged the server with a backup from the 22nd, and were able to reapply our encrypted offsite database backup to the server. We also redid the updates, and obviously made sure perfctl was not present on the backup from the 22nd, which it was not. While we can’t see any evidence that anything touched the piefed database that shouldn’t have, we have changed every single key on the server and in our infrastructure, and we invalidated everyone’s passwords by setting them to null docker compose exec -T db \ psql -U piefed -d piefed -v ON_ERROR_STOP=1 <<‘SQL’ BEGIN; UPDATE public.user SET password_hash = NULL WHERE instance_id = 1 AND password_hash IS NOT NULL; COMMIT; SQL By setting them to null, Piefed very handily forces the user to reset. Our other option was to generate new passwords for everyone, but this felt more intuitive, or at least the least confusing option of two bad options. We also reset the secret key that validates logged in sessions, so this forced everyone to log out too, hence they would have to reset their password. We also redid the firewall, beefing it up an awful lot (unclear if this would have helped with this, but worth it anyway), redid the docker networking away from the piefed default to be more compartmentalised and hopefully secure, and reviewed the whole networking stack. There is still management software, but this is restricted to just a VPN connection and not accessible externally, which has been stress tested - it will fail-secure should the VPN not work. Massive shout of thanks from myself to INeedMana for jumping in and spending hours fixing this with me, guiding me, and putting up with my stress and panic! We will keep monitoring going forwards, and let everyone know if we have any further concerns, but it does look like it is resolved now. Oh and after all that we did manage to update Piefed too :)

I’ve been having a good old play with Lemmy to tweak it to how I want it, but not too much as 1.0 is coming up… soon… Anyway, some of you may have been around here long enough to remember that back at the beginning of 2025 (almost 2 years ago now!) we had to block the UK to comply with the very draconian and only getting worse Online Safety Act (because I am a UK citizen sadly). Mainly, this is because there is no way to prevent children from accessing NSFW content in Lemmy. Now, I’ve been able to add a complete an unremovable block on NSFW content on UK users. Is it ideal? No. Is it something I agree with? Also no. Does it allow us to accept UK users back on Lemmy.zip? Probably…? I need to do more digging, but for now UK users can once again sign up, they just can’t turn on NSFW content. Sorry about that. You’ll have to get your particularly saucy ankle pictures elsewhere. Users in the UK attempting to turn on NSFW will instead see this: Image And UK users going directly to NSFW content will see this instead: Image — I have also recently added a minimum character length to the Lemmy.zip registrations for the answer. You’ve pretty much all filled one in if you’re on Lemmy.zip, but what you don’t see is the amount of people who type stuff such as “lemme in” or “aldkasdkjas” or stuff like that. So, now users have to enter 50 characters to not be auto-rejected. We still read every single one before approving though, we just turn those away that can’t be bothered filling it in. Image — I have also added an improved ban message when a banned user tries to log in. Normal Lemmy just pops up a little message saying can’t log in, now the user gets a full on warning: Image Hopefully you never see that screen, but at least you know it exists now :) — I did also fix .avif images not opening in-line, although this has since been backported by the devs into 0.19.21 so I hope to anything almighty I stop getting complaints about avif images (just need some outdated apps to finally support them!). — I also dropped some very specific local performance improvements into lemmy, including a very detailed and overly complex (for me) bug between postgres and lemmy that was affecting schwim (one our admins) and I’m sure others out there. I will try and explain it to you, while also explaining back to myself. When you scroll to page 2 of a feed like All/New, lemmy says “give me the next 25 posts after the last one I showed.” The way it phrased that to the database was one big combined comparison, roughly “find posts that come after (this pinned-status, this score, this timestamp, this ID)” all bundled into a single tuple comparison. Example tuple comparison: (featured, score, published, id) < (last post’s featured, score, published, id) Here’s where Postgres does the database equivalent of confidently walking into a glass door. When it sees a bundled comparison like that, its query planner (the bit that decides how to actually fetch the rows) only bothers to look at the first column in the tuple to guess how many rows will match. The first column was featured (i.e. is this post pinned?). And the comparison effectively asked for posts where featured < false. Nothing is less than false. False is the bottom. So Postgres looked at that, went “ah, this will match approximately… one row, maybe zero,” and picked a strategy optimised for returning a tiny handful of results. Except it was about to return thousands. So now Postgres is running a plan designed for a teaspoon of data against an ocean of it. To make it worse, lemmy.zip users have community block filters (including ghost-blocks left clinging to communities that were removed, the digital equivalent of a restraining order against someone who already moved away). Those filters nudged the planner toward sorting every single older post in the feed from scratch, every time someone clicked “next page.” The damage: 40 to 100 seconds per page load. Per request. Page two. ~14 GB of temporary files spewed onto disk, because it couldn’t sort that much in memory, so it wrote the overflow to disk and sorted it there. Repeatedly. For a button click. The fix (anticlimactically small, as the best ones are): instead of cramming featured into the bundled comparison and letting Postgres faceplant, we just say up front, “we’re past the pinned posts now, so filter featured = false,” and then only bundle-compare the columns that actually matter (score, time, ID). Same posts. Same order. Postgres now estimates correctly, picks a sane plan, and the results: ~25 milliseconds. Down from up to 100,000. That’s roughly a 4,000× speedup from telling the database something it technically already knew but was too proud to figure out on its own. No migration, no schema change, no new features, just one clause moved so the query planner would stop hallucinating. The downside is if the amount of pinned posts spills over on to page 2, then it wouldn’t show them. However we’ve had at most like 3 pinned posts at once… so that is very unlikely to happen. — For mods, I am also looking at implementing more robust moderation reports for us. Currently admins resolving reports interferes with mods resolving reports, so I am working on a resolution to that, at least until 1.0 is here and changes how reports work. — One more thing I’ve added, once a user does sign up, they should now get this pop up: Image It is an attempt to at least explain how the process works. Hopefully it helps?!