Anyone else notice a long delay for comments/posts from lemmy.world?

Submitted ⁨⁨7⁩ ⁨months⁩ ago⁩ by ⁨jballs@sh.itjust.works⁩ to ⁨main@sh.itjust.works⁩

It might not just be lemmy.world, but several times recently I’ve made posts (usually to !helldivers.lemmy.ca) that I check throughout the day. Then like 8 hours after I’ve made the post, I’ll see comments appear from people on lemmy.world that are several hours old.

It seems like there can be a queue that gets build up and then is suddenly flushed.

I see it happening the other way as well. This morning I made a post from my sh.itjust.works account and then switched over to a lemmy.world account and can’t see it. So the delay is going both ways .

source

Comments

Sort:hotnew top

InEnduringGrowStrong@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
We are currently seeing a backlog in outbound federation towards lemmy.world.
Really, this graph says it all… 📈

Image

I’ve restarted the containers, hoping it starts catching up.

source
- jballs@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
  Oh yeah, that looks pretty telling. Also, just wanted to say thanks for how awesome you guys are on the admin team. The work you guys do and how responsive you are is freaking impressive. Thanks!
  
  source
  - InEnduringGrowStrong@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
    Sadly that line keeps going up.
    I don’t have much of an update but it’s really specific to outbound federation and just with lemmy.world.
    ie: inbound works fine, outbound with other instances works fine.
    
    source
    -> View More Comments
CaptDust@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
I’m experiencing the same and only noticed it when posting to lemmy.world communities. I’d navigate to their instance directly and see both comments that didn’t federate yet and my comment missing. I haven’t observed this on other instances though, lends me to believe this might be just a lemmy.world problem?

source
- jballs@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
  I think I saw somewhere that lemmy.world has grown way bigger than most other instances - probably because the name is less confusing to new users. I’m wondering if they’re running into scaling issues as a result.
  
  source
  - kersploosh@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
    Some of the instance admins have been poking at this lagging federation issue for a few weeks now, trying to figure it out.
    
    The Reddthat admin noticed that Lemmy’s federation process can’t seem to meet demand in some cases. Reddthat has had trouble staying current with lemmy.world due to the network latency between Europe and Australia: https://sh.itjust.works/comment/9807807. I understand that lemmy.nz has seen this, too.
    
    Another thing that has been noticed is spamming of actions from kbin instances. It looks like some process gets stuck in a loop on the kbin side. https://kbin.social/m/kbinEarth@kbin.earth/t/928709/kbin-earth-federation-problems-fixed#entry-comment-5945451
    
    I’m sure there are other contributing problems that still aren’t well understood. This software is a work in progress, after all.
    
    source
    -> View More Comments
InEnduringGrowStrong@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
I’d like to celebrate early and possibly jinx myself in the process but the line is going down.

Image

source
- jballs@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
  Lol the way that’s dropping I’d say a celebration is in order!
  
  source
leftzero@lemmynsfw.com ⁨7⁩ ⁨months⁩ ago
Dbzer0 recently had issues with long federation queues with big instances, due to delays in database access caused by geographic distance between frontend and backend… maybe this might be caused by something similar…?

source
- jballs@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
  Interesting. So by having the database and front end servers in different data centers, they couldn’t keep up with all the requests from the big instances. Sounds like upgrading to Lemmy 0.19.3 changed how the federation queues worked, where it wasn’t a problem before. I wonder if something similar could be going on here.
  
  source
imaqtpie@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
This does happen from time to time where the federation queue gets overwhelmed, usually with Lemmy.world for whatever reason. It appears that our outbound federation queue is really backed up right now.

@InEnduringGrowStrong@sh.itjust.works just restarted the service so it may be working better now.

See our Matrix chat for more details.

source
- jballs@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
  Awesome, you guys rock!
  
  source
verity_kindle@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
I only have a delay when I’m using a VPN.

source
- jballs@sh.itjust.works ⁨7⁩ ⁨months⁩ ago
  Interesting. I’m not using a VPN in my case.
  
  source