Stop right there, criminal scum!
Welcome to Lemmy.zip’s September update. Nobody breaks the law on my watch! I’m confiscating your stolen goods. Now read this update post, or it’s off to jail.
RESIST ARREST <<
THEN PAY WITH YOUR BLOOD
Server updates
I just want to say thank you to everyone who responded to last month’s call for donations. It was a great response, and helps to keep this social media free for everyone.
This month, INeedMana has been busy keeping the server updated and making improvements to the infrastructure, tidying up processes and un-breaking my changes. It all helps to keep the services able to run long-term and efficiently. We even fixed a bug that prevented embeds from working properly!
Throughout August, Lemmy.zip was going swimmingly well up until Sunday 30th, when it decided to fall over in the most obscure way.
Following a slow-down in the UI, a restart of services triggered a latent docker bug. This in turn killed the Lemmy services off, which all started returning 504 errors, or loading the pages very, very slowly (upwards of 60 seconds if it didn’t time out). The API was also down now, so mobile apps stopped working.
After around two hours of debugging, I tried the ultimate fix, one that has never failed me and has always had a 100% success rate. I turned it off and back on again.
Good news, when it restarted we were no longer getting 504 errors! Sadly, we were now getting 502 errors instead. Excusemewhatisgoingon.jpeg
I had solved the pesky 504 error by instead making the UI impossible to access. Thankfully, the API was still up (if still verrrrrrrry sloOoOoOooOooOOOOw) which was both helpful and also completely baffling as to what the issue could be. I had no idea why the error we were seeing before the reboot had completely changed, and suddenly the API that wasn’t working was now working fine.
This ruled out DNS (it’s always DNS), nginx (despite me spending two hours debugging our nginx configs), the postgres database, the pict-rs service, heck I even tried the email service just to be sure.
Somehow, Lemmy was answering API requests, the database was nearly idle, the UI container was actually still serving pages to anyone who could figure out how to get there. Every part of the service was in perfect working order. Except it wasn’t working.
By this point I had eliminated so many possibilities I was drifting towards the only reasonable explanation left: I had personally offended the god(s) of networking.
Suddenly, I noticed something that didn’t quite look right - docker.
More specifically, on the version of docker we were on, it gives each container a MAC address derived from the container’s IP address. At some point, somehow, while trying to work out why the UI wasn’t quite playing nicely, I had triggered a bug that gave two containers in the docker network the same MAC address. Before rebooting, I have no idea which two containers this was, but the error persisted over a reboot. When the server rebooted, every container came back in a different order and got shuffled a new IP, while keeping the MAC address from its old one.
Eventually I narrowed it down to the UI container and the internal proxy. So when the proxy yelled “WHO HAS 172.20.0.6?” into the void, the answer that came back was its own MAC address. It had spent the entire afternoon posting every single page request to itself, then waiting 60 seconds for a reply from a container that was, tragically, also itself.
For once, it wasn’t DNS. It was ARP. You remember ARP exists, right? Because I didn’t.
The fix in the end was to force the lemmy UI off the docker network, and reconnect it with a different fixed IP, which forced docker to derive a fresh MAC (and then restarting the nginx proxy so it picked up the change).
And then everything came back up like nothing had ever been wrong. Fab.
And that’s how I spent a Sunday afternoon.
Please Don’t Be a Lurker!
If you’re new here - WELCOME! I hope you’re enjoying your time here :)
Just one small teeny-tiny request. The greatest gift you can give Lemmy.zip and Piefed.zip isn’t money, praise, or a gif of you pronouncing gif wrong by saying gif instead of gif… maybe. Its participation.
Upvote the things you like. Start a discussion, debate, or just post the same meme of a disappointed cat over and over and over and over and over and over and over and so on… Make new communities if you don’t see one that fits your oddly specific niche obsession. (We don’t judge. Well, we try not to judge anyway.)
The fediverse naturally ebbs and flows, tides of people come and go. But if you’ve found yourself oddly attached to this strange little corner of the internet? Wonderful. Help it breathe. Help it grow. Help it be just a tiny bit weirder in the best possible way.
So if you’ve gone to the effort of clicking Sign Up and proving you’re not a robot (unless you are, in which case hello and welcome to our new AI overlords), then please, I beg of you:
Stick around. Add your voice. It really does make this place better.
Donations
Want to support us? We have a range of donation options to suit you!
Liberpay (Card, Paypal)
Liberpay is our newest donation method. You can donate with Paypal or with a card. The link is liberapay.com/dotzip
OpenCollective (Card)
Our classic donation option, but only supports card payments. The link is opencollective.com/lemmyzip
If you’re enjoying Lemmy.zip or Piefed.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the sites and keeping the virtual lights on.
Ko-Fi (Card, Paypal)
We also have our Ko-Fi page if you’d rather use this site, which also supports Paypal and Card payments.
Le Graph
Piefed.zip
CPU: Image
RAM:
Usage right now: Image
Image storage:
Cloudflare overview:
Requests:
Bandwidth:
Visitors:
Traffic:
Lemmy.zip
CPU:
RAM: Image
Here’s a random graph i thought looked cool:
Image storage:
Cloudflare overview: Image
Requests: Image
Bandwidth:
Visitors: Image
Traffic: Image
Anyway that’s all for this month. I have slightly lost the plot today, so I am now off to bed.
As always, let me know if there’s anything you’d like to see in these updates, always happy to build it in.
Speak soon.
Demigodrick
schwim@piefed.zip
v4ld1z@lemmy.zip
makou@lemmy.zip 4 hours ago
Yo. Am new to Lemmy and I just wanna say this is cool asf. People have a chance to post cool or informative shit without being afraid that a big corpo or anyone will drastically change something that benefits them and makes them profit, whether that would be money or popularity. Its also free, which I have no idea how but we all appreciate it. Thank you