Comment on Downtime - Apologies and what went wrong
Demigodrick@lemmy.zip 1 week ago
I will try and reply to each comment - but you’ve all been really kind and that means so much ❤️
If you’re interested, this graph will show you how far behind we are. We should eventually catch up, but things will likely be very delayed for up to 12 hours.
The status page did not work as expected - and I’ll try and link a few more places where I post updates. If you haven’t yet, definitely join the matrix space and you’ll get minute by minute panic updates 🫠
shortwavesurfer@lemmy.zip 1 week ago
That graph is really kind of neat, but it seems to only be synchronizing with a single instance at a time from what I can tell. I saw the world line has dropped significantly, but the other lines don’t look like they’ve fallen yet.
Demigodrick@lemmy.zip 1 week ago
Yes, the lemmy.world admins kindly manually reset the timer for their instance so it started updating straight away!
If an instance goes down, other instances slowly back off sending retries of activities so not to waste sending them to dead instances.
You can use this tool to see this info. It links lemmy.world but you can search for any instance, and then look up lemmy.zip either under failed or lagging instances. You’ll see on the far right the “next send try” time and date. Looks like a lot will try again around 9pm (although I’m not entirely sure on the timezone there) - so over the next few hours instances will send another try, see that lemmy.zip is back up, and then start federation with us again :)
shortwavesurfer@lemmy.zip 1 week ago
It’s cool to see that there’s logic built-in that keeps instances from sending Federation requests to dead instances. But when an instance comes back online, they will re-synchronize themselves. An instance may drop out of the Federation, but when it comes back, it will get everything it missed. Eventually.