How can I reduce the amount of storage used by my Lemmy instance?

Submitted ⁨⁨11⁩ ⁨months⁩ ago⁩ by ⁨TrinityTek@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

I’ve had a Lemmy instance running on a VPS with 100 GB of storage for a few months and it has filled up. I’ve been searching for ways to reduce the amount of storage used but so far I am coming up empty. Can anyone point me in the right direction?

source

Comments

Sort:hotnew top

bdonvr@thelemmy.club ⁨11⁩ ⁨months⁩ ago
Firstly move pict-rs to object storage. My instance’s pict-rs uses 130GB alone. I pay less than $2/mo to put it on Cloudflare R2. Backblaze B2 might be even cheaper. Instructions: crates.io/crates/pict-rs#filesystem-to-object-sto…

If that doesn’t help enough and you’re comfortable with SQL, you can purge the unnecessary entries in received-activities.

Command: delete from received_activity where published < NOW() - INTERVAL ‘3 days’;

Then do a vacuum full received_activity; to reclaim the space.

This deleted 98 million entries for me and reduced my database size from 49GB to 20GB a week ago when ai started running out of space. No other effect as far as I can tell. Thanks @illecors@lemmy.cafe

source
- TrinityTek@lemmy.world ⁨11⁩ ⁨months⁩ ago
  Thank you, that SQL command looks like exactly what I’m after! I’m going to give that a shot. I appreciate the help!
  
  source
- MrRazamataz@lemmy.razbot.xyz ⁨11⁩ ⁨months⁩ ago
  what is stored in received_activity ? anything important? I mean obviously it’s something the instance has received from other instances but is this then stored somewhere else (like comments then stored elsewhere for eg)
  
  source
  - bdonvr@thelemmy.club ⁨11⁩ ⁨months⁩ ago
    As far as I’ve been told it’s basically just a log of all received activities. Nothing references it.
    
    source
  - Die4Ever@programming.dev ⁨11⁩ ⁨months⁩ ago
    in v0.19.0 Lemmy will automatically delete entries over 7 days old
    
    github.com/LemmyNet/lemmy/issues/4113
    
    github.com/…/cb01427dcff14b3d88e30220695fc9797878…
    
    source
scrubbles@poptalk.scrubbles.tech ⁨11⁩ ⁨months⁩ ago
Instead of using the hard drive for pictrs, I suggest using it’s S3 capabilities and migrating to bucket based storage. You’ll save way more money and keep the expensive VPS hard drive just doing text and DB things. I think I spend maybe a dollar a month in S3 storage.

source
- nix@merv.news ⁨11⁩ ⁨months⁩ ago
  Is there a guide for switching over?
  
  source
  - bdonvr@thelemmy.club ⁨11⁩ ⁨months⁩ ago
    crates.io/crates/pict-rs#filesystem-to-object-sto…
    
    source
- TrinityTek@lemmy.world ⁨11⁩ ⁨months⁩ ago
  Thanks for the suggestion! As Nix asked, do you happen to know of a guide or any documentation I could reference for this?
  
  source
  - scrubbles@poptalk.scrubbles.tech ⁨11⁩ ⁨months⁩ ago
    I used the same one @bdonvr@thelemmy.club used - crates.io/crates/pict-rs#filesystem-to-object-sto…
    
    source
hitagi@ani.social ⁨11⁩ ⁨months⁩ ago
pict-rs has the option to compress images. Ours is set to WEBP with 1280 pixels either side max.

source
- TrinityTek@lemmy.world ⁨11⁩ ⁨months⁩ ago
  That sounds like a good idea. Do you know of any documentation for this? I’d like to give it a try.
  
  source
  - hitagi@ani.social ⁨11⁩ ⁨months⁩ ago
    git.asonix.dog/asonix/pict-rs/src/…/pict-rs.toml
    
    source
lemmy@linkopath.com ⁨11⁩ ⁨months⁩ ago
I ran into the same problem and ended up switching to an S3 with Vultr. It’s been a while since I did it but here are the links that I used to figure it out. I’m deployed using Lemmy-Easy-Deploy.

I used a combination of:

lemmy.world/post/538280

github.com/…/ADVANCED_CONFIGURATION.md

git.asonix.dog/asonix/pict-rs/#user-content-files…

Good luck!

source
- TrinityTek@lemmy.world ⁨11⁩ ⁨months⁩ ago
  Wow, thank you for these great resources! I will check it out. I really appreciate it!
  
  source
- jeena@jemmy.jeena.net ⁨11⁩ ⁨months⁩ ago
  So I wanted to try it and found that Synology offers 15 GB for free so I tried it and it was extreamly easy to do. Set up a bucket and then fill in the ENV variables in docker-compose restart and it works, impressive. I might really need to think getting a good deal and doing that for my other fediverse stuff like Mastodon, Peertube and Matrix too, I think it’s equaly easy there.
  
  source
  - jeena@jemmy.jeena.net ⁨11⁩ ⁨months⁩ ago
    Ah, but all my old pictures are not available anymore …`
    
    source
    -> View More Comments
Die4Ever@programming.dev ⁨11⁩ ⁨months⁩ ago
you could try disabling pictrs caching github.com/LemmyNet/lemmy/pull/3897/files

# Cache remote images cache_remote_images: false

I don’t think that will clear the existing cache though
source
- Dave@lemmy.nz ⁨11⁩ ⁨months⁩ ago
  Isn’t this only on the as yet unreleased version 0.19?
  
  Asking as someone with a 165GB picts cache…
  
  source
  - Die4Ever@programming.dev ⁨11⁩ ⁨months⁩ ago
    yes you’re right, also that config flag was renamed to cache_external_link_previews
    
    source
ramble81@lemm.ee ⁨11⁩ ⁨months⁩ ago
I love how OP doesn’t say it, but everyone immediately goes “it’s the pictures”

source
- Scrath@feddit.de ⁨11⁩ ⁨months⁩ ago
  Honestly, what else would it be? Text takes ridiculously little storage compared to a single picture of a decent resolution.
  
  source
Shadow@lemmy.ca ⁨11⁩ ⁨months⁩ ago
Are you on the latest version?

Is the storage pictrs, or your db?

source
- TrinityTek@lemmy.world ⁨11⁩ ⁨months⁩ ago
  Sadly, no. My server has been a bit neglected but it’s been plugging along and working fine for the most part. I need to upgrade though. And I assume pictrs but to be honest I haven’t checked. I just noticed today it was running poorly and checked and the drive is full.
  
  source
  - Shadow@lemmy.ca ⁨11⁩ ⁨months⁩ ago
    The last release (or maybe the one before) did a bunch of DB cleanup that reduces the db size by about 20gb.
    
    source