Backing up across machines with low free space?

Submitted ⁨⁨2⁩ ⁨weeks⁩ ago⁩ by ⁨tofu@lemmy.nocturnal.garden⁩ to ⁨selfhosting@slrpnk.net⁩

I run some services on machine A. I want to backup the data to machine B. It’s around 40 gigabytes in thousands of small files. I would prefer to compress the data, but there’s not enough space on machine A to keep the data twice (original + tar archive). I would like to avoid copying that many files via scp/rsync since it’s taking forever and a bit fragile.

Any solutions on how to solve this? Is it possible to compress live on machine A and stream into the archive on machine B without the need to keep the big archive file on A?

Eventually, it’s supposed to be automated and B has like the last 3 dailys, 1 weekly, 1 montly.

B can ssh into A.

source

Comments

Sort:hotnew top

thelittleblackbird@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
Borgbackup is all you need, you even don’t need to allocate the space in your A machine…

source
- tofu@lemmy.nocturnal.garden ⁨2⁩ ⁨weeks⁩ ago
  Seems like Borg Backup Pull Mode does what I need, using sshfs:
  
  borgbackup.readthedocs.io/en/…/pull-backup.html
  
  source
  - thelittleblackbird@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Certainly it is a good way to go, but in my experience try the usual work flow, it is way more stable and the cpu burden is share between both nodes -> faster backups
    
    source
    -> View More Comments
qaz@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
Perhaps Kopia? It supports compression and deduplication.

source
- tofu@lemmy.nocturnal.garden ⁨2⁩ ⁨weeks⁩ ago
  Thanks, I’ll have a look!
  
  source
fruitycoder@sh.itjust.works ⁨2⁩ ⁨weeks⁩ ago
Have the filesystem compress by default and ship that?

I know you can do it with zfs. Export a snapshot of a fs with compression enabled. I’m less sure if you can do it with a btrfs, lvm, or xfs though. I know they support fs level compression though.

source
- tofu@lemmy.nocturnal.garden ⁨2⁩ ⁨weeks⁩ ago
  One machine is a cloud VM with LVM I think, the other is my NAS with zfs. I think with zfs to zfs it would easily work!
  
  source
tofu@lemmy.nocturnal.garden ⁨2⁩ ⁨weeks⁩ ago
This seems promising

tar streaming

source
redlemace@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
To compress you need diskspace. Simple as that. Three daily from 40 Gb in small files? By thor, sure hope you think incrementals. Have you tried backupc?

But honestly, it’s worth -and most reliable- to invest in a new bigger disk to solve the problem instead of working your way around it.

source
- tofu@lemmy.nocturnal.garden ⁨2⁩ ⁨weeks⁩ ago
  It’s a VPS and I don’t want to increase the monthly cost - in fact I want to decrease it by disabling the backups the hoster offers. Machine B is my NAS at home.
  
  Of course, incremental would be preferable, but I’m not sure how that would work in that constellation.
  
  I wonder if I can compress to B directly. Maybe I could mount a volume from B into A via wireguard and nfs, now that I think about it.
  
  source
  - redlemace@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Do you have unlimited data transfer with your vps? Transferring 40Gb three times each and every day is quite some traffic.
    
    Why not host it at home then? I bought a used lenovo thinkcenter (m720q on top of my head) for 75€ It uses only 35 watt, it has 750 Gb storage and I only added some RAM.
    
    I pay 1€ per month for a fixed public ip to avoid ddns. My NAS is at a family member (over vpn to home) for offsite backups.
    
    source
    -> View More Comments