Comment on Wondering if running a single user Lemmy is an overkill
erick@piefed.erick.sh 5 days agoAh, good catch. This is something I have to look into. Other self-hosted apps I have usually keep a local cache for a few days only and fetch on demand when needed. Need to explore if both Lemmy and PieFed to something similar.
squaresinger@lemmy.world 5 days ago
Lemmy fetches everything that has ever been posted in any community that any user on that instance is subscribed to and keeps it indefinitely.
Since most activity happens in big communities that most people are subscribed to, most instances keep full, persistent copies of most things that were ever posted to lemmy.
That’s why Lemmy scales so badly. If Lemmy was the size of Reddit, every instance would have to have storage capacity in the same order of magnitude as all of Reddit itself.
The problem only gets worse with time, since all that has been posted still remains.
The total replication also means that the copies need to be moderated by every instance individually, since every instance stores a copy of everything. So if e.g. someone posts illegal content on another instance and your instance stores a replica, you are just as legally liable for that illegal content as the original instance. Thus you have to moderate everything that runs over your instance.
Moderation effort is thus also replicated across all instances.
That bad scaling in storage and moderation is btw the reason why e.g. lemm.ee shut down. It was just too much cost and work to keep the instance running.
erick@piefed.erick.sh 5 days ago
As far as I can tell PieFed already handles deleting old content (1 week by default, but I’m looking at the code on my phone so not the best way of doing research). I’ll do some more code reading later if I have a chance.