Comment on Spreading of the 100 biggest Lemmy communities
Dave@lemmy.nz 2 months agoWe are fine, but it’s not fixed. I have a second VPS running in Finland, using this queue batcher. The Lemmy.world team kindly set up their server to point to this VPS instead of the actual Lemmy.nz server, then the VPS collects all the events and sends them to the Lemmy.nz server in batches of 100.
It keeps us up to date, but it’s cheating 😆
Last I heard Aussie.zone doesn’t have this setup, but they do have a prefetcher (or rather, Nothing4You, who made the queue batcher, is running a prefetcher for them). This basically takes the new comments and posts from Lemmy.world, and sends a request to Aussie.zone to fetch that post. Because this happens outside the normal federation queue it can be done in parallel. It means when Aussie.zone receives the federated activity from Lemmy.world, it already has it, so it can reply quicker and process more events per second. Lemmy clears out activities older than a week in a weekly scheduled job, which is why you will see Aussie.zone’s backlog drop a bit once a week. They won’t get that content from Lemmy.world, it’s just lost. Because of the prefetcher, it’s likely just up/down votes (which can’t be prefetched).
Blaze@sopuli.xyz 2 months ago
Very interesting, thanks!