Backup/Server Options - is Syncthing / Nextcloud really the go?

Submitted ⁨⁨3⁩ ⁨weeks⁩ ago⁩ by ⁨oneser@lemmy.zip⁩ to ⁨selfhosted@lemmy.world⁩

Unfortunately, as Lemmy doesn’t appear to have an indexed search function I am not sure if this is a common question or not. Please point me to the necessary thread/s if this is answered elsewhere.

I’m looking to continue the ball rolling on my home server. Jellyfin setup was a nice dive back into networking, which I haven’t done for quite some time and the logical next step is to get all the data we want to retain into a single hub.

Most searches seem to point to syncthing with nextcloud, but before I get started, I want to check I am really going in the right direction.

I would like to primarily remove the space burden from my devices and dump them all onto a few drives and a cloud backup (in case of physical loss of drives). I believe syncthing can do this, but some appear to say that it is not an effective archive tool.

I would like to be able to retrieve this data without much hassle for e.g. photo editing, and place the finished file back on the server. Preferably the local copy would be removed again, to reduce the need for extra space on each device. I would like to run this over nextcloud, but might be misunderstanding the software a bit.

source

Comments

Sort:hotnew top

Onomatopoeia@lemmy.cafe ⁨3⁩ ⁨weeks⁩ ago
As others have said, sync isn’t backup.

It may be part of a backup plan, however.

I use Syncthing on my mobile devices to keep data created on the devices synchronized to my server at home. Things like photos sync to home over any connection, while I sync other stuff only over wifi. Syncthing-Fork allows you to set these conditions on a per-folder-pair basis.

That server becomes my authoritative box for any data. All that data is then mirrored on a schedule to 2 other systems at home (a NAS and a large drive on another box).

The main server also has a cloud backup which runs continuously.

So I have 3 local copies of data to recover from if I have a hardware failure, and a cloud backup.

I find tools like Syncthing and Resilio are good for synchronization, especially mobile devices. But between full-pc-OS devices, I just use native tools (scripts and schedules) because I don’t want synchronization, but specific patterns of copying/mirroring, etc.

I do use Resilio for ad-hoc access to almost any file on my server, since it’s Conditional Sync feature permits me to connect with a mobile device from anywhere and sync only the selected files. So I can grab a movie or TV show, Resilio will sync it and I can watch it once the sync is complete.

source
savvywolf@pawb.social ⁨3⁩ ⁨weeks⁩ ago
Syncing software is not a backup. I’ve had cases where they get confused and end up deleting data. They’ll also blindly copy over corrupted or randomwared files.

source
- Hule@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  Randomware should be a thing!
  
  source
  - rumba@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
    It was, shareware. 10000 games on this CD!!!
    
    source
communism@lemmy.ml ⁨3⁩ ⁨weeks⁩ ago
You’ve not made it clear what exactly it is you want. Nextcloud or syncthing are good for syncing personal files. If you want to make server backups, they’re not gonna be the way.

If you want to automate backups, you could just use a cronjob to make a tarball and rsync it?

source
- GenderNeutralBro@lemmy.sdf.org ⁨3⁩ ⁨weeks⁩ ago
  Better yet, use borg to back up. Managing your own tars is a burden. Borg does duduplication, encryption, compression, and incrementals. It’s as easy to use as rsync but it’s a proper backup tool, rather than a syncing tool.
  
  Not the only option, but it’s open source, and a lot of hosts support it directly. Also works great for local backups to external media. Check out Vorta if you want a GUI.
  
  source
philpo@feddit.org ⁨3⁩ ⁨weeks⁩ ago
Syncthing and nextcloud are not a good backup solution. Like ever. Potentially they aren’t even a backup solution at all. Or even cause data loss.

You sadly didn’t tell us too much about what you are actually trying to backup and how your infrastructure looks like.

If I understand you correctly you want to centralise the files that are currently hosted on a diverse set of devices into a central file storage on your server and backup from there. Right? That’s a fair goal and something I absolutely do myself - and both NextCloud as well as syncthing will help you make the files accessible for devices.

Now,back to the backup part.

You want basically three things from backup: They need to reliable (doesn’t help when you can’t access your files anymore because they are corrupted), you want them to be as unaffected by any potential risks as possible and let’s face it,you probably want them cheap. The second part basically dictates that for an online backup you want something that can do versioning so corrupted data (e.g. from ransomware) is not simply written over.

My current approach is: I have an internal backup server (see below), an external backup in the cloud, and a cold storage backup in a bank safe. Sounds like a lot? We will see.

Let’s look at cloud storage first. There are a multitude of solutions available for free with Duplicati, urBackup or goMFT being some fairly popular ones - I personally use Duplicati. These periodically scan the folders for changes, encrypt the files and send them to a cloud provider of your choice (e.g. an S3 bucket.) and to some extent can also do the versioning. (Although it’s safer to regulate that via a bucket policy as otherwise the application needs delete rights - which means in theory could delete all the data when compromised). Main benefit is the ease of access - you need to restore a single file? Done fast and easy. Not so much for a whole setup, restoring things can get quite expensive.

If you use ZFS there is also the option to use ZFS sent to backup, but as there is currently no reliable European Union ZFS sent provider I am aware of (rsync.net does this,but is US based) legally cannot use them. So no experience on that.

To backup clients completly and VMs/LXC it might also make sense to use a designated backup server,e.g. the proxmox backup server. These do require local (as in “where the PBS is running” storage, though, so a local PBS and a cloud storage behind doesn’t work. (There is a “hosted PBS” Service available, though from Tuxis. They work really well). But it can make sense to let a zimablade run a few old hard drives for a few hours a day for that.

For offsite and online backup - as a full restore is always expensive and time consuming from the cloud- I also use two USB hard drives. One is always stored in a locker in a bank vault and every few months I change drive - so in case of a full server loss I only would need to restore the state of a (at max) 4 month old server via USB and then update stuff from the cloud for the 4 months after that.

Now, to be extra sure I also burn the most important files (documents about the house,insurances,degrees,financial and tax data, healthcare records, photos of lifetime events, e.g. weddings, birthdays,births, graduations as well as “emergency data restore howtos”, password files, basically all the stuff I want to make sure my heirs/kids have access to if I die) on blue archive (important, not normal disks!) M-Discs. They are supposed to last far longer than normal blue rays and most consumer accessible media. These are stored locally,in the safe and at the court that holds our will. The reasons for that? Powered off hard drives lose data quite fast and if the wife and I perish at the same time, eg. because we have a car crash or the house burns down the issue is time: Cloud backup might not be available anymore as our bank accounts are frozen and therefore the backup is no longer paid for. The bank safe is not accessible for a long time for the same reason. When someone then accesses the USV drive it might be of no use. The server might be powered off or damaged. And sadly the legal system here can take years (up to 7 years are my planning times) before they can actually access the data.

source
Dave@lemmy.nz ⁨3⁩ ⁨weeks⁩ ago
Remember sync isn’t a good backup. You’re thinking of loss of drives but if this is important data you need to also consider mistakes.

If you accidentally delete files you shouldn’t, you don’t want this deletion to sync to all your copies so it’s gone for good and the backup doesn’t help.

Personally I use borgmatic to keep incremental, deduplicated backups. Then I can go back to previous states.

If you install nextcloud all in one, it comes with a backup solution (also borg based). Then devices don’t need a copy of every file. But you’ll want your server to have a backup drive for this.

I then sync my borg backup to a backblaze b2 bucket for offsite, encrypted backup using rclone. That then meets the 3 2 1 backup plan.

I notice you mention Jellyfin. I don’t back up my Jellyfin media, the cloud storage for that could get very expensive and I could get it again if I needed it.

source
- merde@sh.itjust.works ⁨3⁩ ⁨weeks⁩ ago
  
  If you accidentally delete files you shouldn’t, you don’t want this deletion to sync to all your copies so it’s gone for good and the backup doesn’t help.
  
  edit folder > ignore delete
  
  and you don’t have to worry about syncthing deleting your backup
  
  source
  - aksdb@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
    That, on the other hand, is only viable, if you are sure, data never needs to expire. Dedicated backup solutions work with retention policies.
    
    source
  - Dave@lemmy.nz ⁨3⁩ ⁨weeks⁩ ago
    Yip you can do that but then it’s messy! And what if you overwrite a file by accident?
    
    And if you do lose your hard drive then you have a weird state to restore from.
    
    I’d much prefer the ability to restore to a point in time that comes with something like borg.
    
    source
    -> View More Comments
- Creat@discuss.tchncs.de ⁨3⁩ ⁨weeks⁩ ago
  Just to add to this, if you have periodic snapshots on the server side, this does solve the problem. And it simplifies things a lot.
  
  source
  - Dave@lemmy.nz ⁨3⁩ ⁨weeks⁩ ago
    Yes, if you go with something like syncthing, have it also sync to a server where you run borg backup so you get the incremental backup.
    
    source
junkthief@lemmy.blahaj.zone ⁨3⁩ ⁨weeks⁩ ago
Lemmy definitely has a search function. Due to federation (and specifically how lemmy.zip has federated with lemmy.world) you can easily search from your home instance of lemmy.zip and limit your search to this community

…blahaj.zone/…/fc1f7747-1258-471d-8db0-4f0bbc77e7…

source
null_dot@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
Adding to what everyone else has already said, you want sync and backup.

Sync to a central location and backup from there.

For sync, you want syncthing or nextcloud. I would lean towards syncthing for media. If you had a million files in a complex folder structure and a dozen users with different access requirements and instant sync and collision protection is important then nextcloud might be the go. Otherwise syncthing is much more manageable.

My recommendation with syncthing, which is not obvious, is to set up a single hub which each client syncs with. By default you end up with a mesh where everything is connected to everything. It’s very difficult to manage with a lot of folders and devices. Turn off discovery and input the server / hub details manually.

For backup, if you have a lot of media you want deduplication. If yesterday’s backup included ABC and today’s is ABCD you only want to transfer D. This is similar to an incremental backup, but the subtle difference is that with deduplication the most recent backup is the “full backup” with the “diffs” going backwards in time, allowing you to purge old backups. I like borgmatic but there are others.

I would also consider carefully exactly what is worth backing up on what service. I don’t backup movies and tv series at all.

My final recommendation is, it’s critically important to test deploying your backups regularly.

source
KaninchenSpeed@sh.itjust.works ⁨3⁩ ⁨weeks⁩ ago
I’m currently doing phone backups to a samba smb server with an app called smbsync2 from fdroid. It can copy or move files and dorectories from and to a smb share on a schedule.

To get the archiving functionality you can do automatic ZFS snapshots. You can restore the entire snapshot or mount it via the terminal, or samba apparenly has a feature to display files from snapshots as shadow copies on windows.

There is one limitation of this is, if a client deletes a file on the local storage, the file on the server doesn’t get deleted automaticly.

source
- Sir_Kevin@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
  I second smbsync2 for phone backup. It’s very customizable and once setup, just works.
  
  I also use syncthing but only for syncing files I want to be the same on all devices. FreeTube watch history and subscriptions, stuff like that.
  
  source
- Onomatopoeia@lemmy.cafe ⁨3⁩ ⁨weeks⁩ ago
  One advantage of Syncthing over smbsync is it can use any network connection, not just LAN.
  
  You can configure to only use LAN, or even specific LANs if needed (identified by SSID). I have a few sync jobs that work this way.
  
  source
rumba@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
3 copies of your data (original and 2 backups)

2 different media. (pick 2: Raid, jbod, ssd, tape, managed cloud)

1 off-site

I’m gonna add in: one offline

So

original data on a mirror raid or raid 5

External disc disconnected backup trued up whenever you can

Syncthing/rsync/b2/S3 whatevs to get it off-site.

The nice part about syncthing is it will stop if a certain percentage of you data changes. It has a good chance to stop ransomware as long as it’s not too slow at the encryption.

source
napkin2020@sh.itjust.works ⁨3⁩ ⁨weeks⁩ ago
Exactly what are you trying to back up? If it’s for something like, let’s say, a password manager, syncing is the worst way imaginable.

But if it’s for your Jellyfin media backup, rsync is pretty sufficient imo. They rarely change, costs pretty penny to back up and (relatively) easy to rebuild.

So it all comes down to what your scenario is.

source