Comments

• calliope@retrolemmy.com ⁨6⁩ ⁨months⁩ ago

  Tangentially, I don’t see people talk about rclone a lot, which is like rsync for cloud storage.

  It’s awesome for moving things from one provider to another, for example.

  source

  • Landless2029@lemmy.world ⁨6⁩ ⁨months⁩ ago

    I tried rclone once because I wanted to sync a single folder from documents and freaked out when it looked like it was going to purge all documents except for my targeted folder.

    Then I just did it via the portal…

    source

    • calliope@retrolemmy.com ⁨6⁩ ⁨months⁩ ago

      rsync can sometimes look similarly scary! I very clearly remember triple-checking what it’s doing.

      rclone works amazingly well if you have hundreds of folders or thousands of files and you can’t be bothered to babysit a portal.

      source
  • Eldritch@piefed.world ⁨6⁩ ⁨months⁩ ago

    It's fine. But yes in the Linux space. We tend to want to host ourselves. Not have to trust some administrator of some cloud we don't know/trust.

    source

    • TehNomad@piefed.social ⁨6⁩ ⁨months⁩ ago

      rclone does support other protocols besides S3. You can also selfhost your own S3 storage.

      source
    • calliope@retrolemmy.com ⁨6⁩ ⁨months⁩ ago

      [deleted]

      source

      • -> View More Comments
• dohpaz42@lemmy.world ⁨6⁩ ⁨months⁩ ago

  Here’s how I approach old and slow:

  1. Older software is mature and battle tested. It’s been around long enough that the developers should know what they’re doing, and have built a strong community for help and support.
  2. Slow is okay when it comes to accuracy. Would I love to back up my gigabytes (peanuts compared to some of you folks out there with data centers in your attics) in seconds? Yes. But more importantly, I’d rather have my data be valid for if I ever need to do any kind of restore. And I’ve been around the block enough times in my career to see many useless backups.

  source
• tomkatt@lemmy.world ⁨6⁩ ⁨months⁩ ago

  Rsync is great. I’ve been using it to back up my book library from my local Calibre collection to my NAS for years, it’s absurdly simple and convenient. Plus, -ruv lets me ignore unchanged files and backup recursively, and if I clean up locally and need that replicated, just need to add —delete.

  source
• NuXCOM_90Percent@lemmy.zip ⁨6⁩ ⁨months⁩ ago

  I would generally argue that rsync is not a backup solution. But it is one of the best transfer/archiving solutions.

  Yes, it is INCREDIBLY powerful and is often 90% of what people actually want/need. But to be an actual backup solution you still need infrastructure around that. Bare minimum is a crontab. But if you are actually backing something up (not just copying it to a local directory) then you need some logging/retry logic on top of that.

  At which point you are building your own borg, as it were. Which, to be clear, is a great thing to do. But… backups are incredibly important and it is very much important to understand what a backup actually needs to be.

  source

  • Colloidal@programming.dev ⁨6⁩ ⁨months⁩ ago

    Borg gang represent!

    source
  • non_burglar@lemmy.world ⁨6⁩ ⁨months⁩ ago

    I use rsync and a pruning script in crontab on my NFS mounts. I’ve tested it numerous times breaking containers and restoring them from backup. It works great for me at home because I don’t need anything older than 4 monthly, 4 weekly, and 7 daily backups.

    However, in my job I prefer something like bacula. The extra features and granularity of restore options makes a world of difference when someone calls because they deleted prod files.

    source

    • tal@olio.cafe ⁨6⁩ ⁨months⁩ ago

      I don’t know if there’s a term for them, but Bacula (and I think AMANDA might fall into this camp, but I haven’t looked at it in ages) are oriented more towards…"institutional” backup. Like, there’s a dedicated backup server, maybe dedicated offline media like tapes, the backup server needs to drive the backup, etc).

      There are some things that rsnapshot, rdiff-backup, duplicity, and so forth won’t do.

      • At least some of them (rdiff-backup, for one) won’t dedup files with different names. If a file is unchanged, it won’t use extra storage, but it won’t identify different identical files at different locations. This usually isn’t all that important for a single host, other than maybe if you rename files, but if you’re backing up many different hosts, as in an institutional setting, they likely files in common. They aren’t intended to back up multiple hosts to a single, shared repository.

      • Pull-only. I think that it might be possible to run some of the above three in “pull” mode, where the backup server connects and gets the backup, but where they don’t have the ability to write to the backup server. This may be desirable if you’re concerned about a host being compromised, but not the backup server, since it means that an attacker can’t go dick with your backups. Think of those cybercriminals who encrypt data at a company and wipe other copies and then demand a ransom for an unlock key. But the “institutional” backup systems are going to be aimed at having the backup server drive all this, and have the backup server have access to log into the individual hosts and pull the backups over.

      • Dedup for non-identical files. Note that restic can do this. While files might not be identical, they might share some common elements, and one might want to try to take advantage of that in backup storage.

      • rdiff-backup and rsnapshot don’t do encryption (though duplicity does). If one intends to use storage not under one’s physical control (e.g. “cloud backup"), this might be a concern.

      • No “full” backups. Some backup programs follow a scheme where one periodically does a backup that stores a full copy of the data, and then stores “incremental” backups from the last full backup. All rsnapshot, rdiff-backup, and duplicity are always-incremental, and are aimed at storing their backups on a single destination filesystem. A split between “full” and “incremental” is probably something you want if you’re using, say, tape storage and having backups that span multiple tapes, since it controls how many pieces of media you have to dig up to perform a restore.

      • I don’t know how Bacula or AMANDA handle it, if at all, but if you have a DBMS like PostgreSQL or MySQL or the like, it’s may be constantly receiving writes. This means that you can’t get an atomic snapshot of the database, which is critical if you want to be reliably backing up the storage. I don’t know what the convention is here, but I’d guess either using filesystem-level atomic snapshot support (e.g. btrfs) or requiring the backup system to be aware of the DBMS and instructing it to suspend modification while it does the backup. rsnapshot, rdiff-backup, and duplicity aren’t going to do anything like that.

      I’d agree that using the more-heavyweight, “institutional” backup programs can make sense for some use cases, like if you’re backing up many workstations or something.

      source
  • tal@olio.cafe ⁨6⁩ ⁨months⁩ ago

    I would generally argue that rsync is not a backup solution.

    Yeah, if you want to use rsync, you’re probably better-off using something like rdiff-backup, which makes use of rsync to generate backups and store them efficiently.

    rsync: one-way synchronization

    unison: bidirectional synchronization

    git: synchronization of text files with good interactive merging.

    rdiff-backup: rsync-based backups. I used to use this and moved to restic, as the backupninja target for rdiff-backup has kind of fallen into disrepair.

    source

    • koala@programming.dev ⁨6⁩ ⁨months⁩ ago

      Beware rdiff-backup. It certainly does turn rsync (not a backup program) into a backup program.

      However, I used rdiff-backup in the past and it can be a bit problematic. If I remember correctly, every “snapshot” you keep in rdiff-backup uses as many inodes as the thing you are backing up. (Because every “file” in the snapshot is either a file or a hard link to an identical version of that file in another snapshot.) So this can be a problem if you store many snapshots of many files.

      But it does make rsync a backup solution; a snapshot or a redundant copy is very useful, but it’s not a backup.

      (OTOH, rsync is still wonderful for large transfers.)

      source

      • -> View More Comments
    • melfie@lemy.lol ⁨6⁩ ⁨months⁩ ago

      Having a synced copy elsewhere is not an adequate backup and snapshots are pretty important. I recently had RAM go bad and my most recent backups had corrupt data, but having previous snapshots saved the day.

      source

      • -> View More Comments
    • neidu3@sh.itjust.works ⁨6⁩ ⁨months⁩ ago

      +1 for rfiff-backup. Been usinit for 20 years or so, and I love it.

      source
• i_stole_ur_taco@lemmy.ca ⁨6⁩ ⁨months⁩ ago

  The thing I hate most about rsync is that I always fumble to get the right syntax and flags.

  This is a problem because once it’s working I never have to touch it ever again because it just works and keeping working. There’s not enough time to memorize the usage.

  source

  • oddlyqueer@lemmy.ml ⁨6⁩ ⁨months⁩ ago

    This is why I still don’t know sed and awk syntax lol. I eventually get the data in the shape I need and then move on, and never imprint how they actually work. Still feel like a script kiddie every time I use them (so once every few years).

    source

    • tal@olio.cafe ⁨6⁩ ⁨months⁩ ago

      sed can do a bunch of things, but I overwhelmingly use it for a single operation in a pipeline: the s// operation. I think that that’s worth knowing.

      sed 's/foo/bar/' will replace all the first text in each line matching the regex “foo” with “bar”.

      That’ll already handle a lot of cases, but a few other helpful sub-uses:

      sed 's/foo/bar/g' will replace all text matching regex “foo” with “bar”, even if there are more than one per line

      `sed ‘s/([0-9a-f]*)/0x\1/g’ will take the text inside the backslash-escaped parens and put that matched text back in the replacement text, where one has ‘\1’. In the above example, that’s finding all hexadecimal strings and prefixing them with ‘0x’

      If you want to match a literal “/”, the easiest way to do it is to just use a different separator; if you use something other than a “/” as separator after the “s”, sed will expect that later in the expression too, like this:

      `sed ‘s%/%SLASH%g’ will replace all instances of a “/” in the text with “SLASH”.

      source
  • NuXCOM_90Percent@lemmy.zip ⁨6⁩ ⁨months⁩ ago

    One trick that one of my students taught me a decade or so ago is to actually make an alias to list the useful flags.

    Yes, a lot of us think we are smart and set up aliases/functions and have a huge list of them that we never remember or, even worse, ONLY remember. What I noticed her doing was having something like goodman-rsync that would just echo out a list of the most useful flags and what they actually do.

    So nine times out of 10 I just want rsync -azvh --progress ${SRC} ${DEST} but when I am doing something funky and am thinking “I vaguely recall how to do this”? dumbman rsync and I get a quick cheat sheet of what flags I have found REALLY useful in the past or even just explaining what azvh actually does. And I just keep that in the repo of dotfiles I copy to machines I work on regularly.

    source

    • JohnAnthony@lemmy.dbzer0.com ⁨6⁩ ⁨months⁩ ago

      rsync -avzhP gang unite! I knew someone would have posted my standard flags

      source
    • muix@lemmy.sdf.org ⁨6⁩ ⁨months⁩ ago

      tldr and atuin have been my main way of remembering complex but frequent flag combinations

      source

      • -> View More Comments
    • tal@olio.cafe ⁨6⁩ ⁨months⁩ ago

      Most Unix commands will show a short list of the most-helpful flags if you use --help or -h.

      source
  • mesamunefire@piefed.social ⁨6⁩ ⁨months⁩ ago

    I feel this too. I have a couple of “spells” that work wonders in a literal small notebook with other one liners over the years. Its my spell book lol.

    source
• solrize@lemmy.ml ⁨6⁩ ⁨months⁩ ago

  I’ve been using borg because of the backend encryption and because the deduplication and snapshot features are really nice. It could be interesting to have cross-archive deduplication but maybe I can get something like that by reorganizing my backups.

  source

  • cmgvd3lw@discuss.tchncs.de ⁨6⁩ ⁨months⁩ ago

    I think Arch wiki recommends rsync for backups

    source
• mesamunefire@piefed.social ⁨6⁩ ⁨months⁩ ago

  Ive personally used rsync for backups for about….15 years or so? Its worked out great. An awesome video going over all the basics and what you can do with it.

  source

  • eager_eagle@lemmy.world ⁨6⁩ ⁨months⁩ ago

    It works fine, my issue with that it’s just not efficient. If you want a “time travel” feature, your only option is to duplicate data. There, compression, and encryption for off-site backups is where other tools shine.

    source

    • suicidaleggroll@lemmy.world ⁨6⁩ ⁨months⁩ ago

      If you want a “time travel” feature, your only option is to duplicate data.

      Not true. Look at the --link-dest flag. Encryption, sure, rsync can’t do that, but incremental backups are fine and compression I handle at the filesystem level anyway.

      source

      • -> View More Comments
    • bandwidthcrisis@lemmy.world ⁨6⁩ ⁨months⁩ ago

      I have it add a backup suffix based on the date. It moves changed and deleted files to another directory adding the date to the filename.

      It can also do hard-link copied so that you can have multiple full directory trees to avoid all that duplication.

      No file deltas or compression, but it does mean that you can access the backups directly.

      source

      • -> View More Comments
    • nekusoul@lemmy.nekusoul.de ⁨6⁩ ⁨months⁩ ago

      Agree. It’s neat for file transfers and simple one-shot backups, but if you’re looking for a proper backup solution other tools/services have advanced virtually every aspect of backups it pretty much always makes sense to use one of those instead.

      source
  • confusedpuppy@lemmy.dbzer0.com ⁨6⁩ ⁨months⁩ ago

    I use rsync for many of the reasons covered in the video. It’s widely available and has a long history. To me that feels important because it’s had time to become stable and reliable. Using Linux is a hobby for me so my needs are quite low. It’s nice to have a tool that just works.

    I use it for all my backups and moving my backups to off network locations as well as file/folder transfers on my own network.

    I even made my tool (codeberg.org/taters/rTransfer) to simplify all my rsync commands into readable files because rsync commands can get quite long and overwhelming. It’s especially useful chaining multiple rsync commands together to run under a single command.

    I’ve tried other backup and syncing programs and I’ve had bad experiences with all of them. Other backup programs have failed to restore my system. Syncing programs constantly stop working and I got tired of always troubleshooting. Rsync when set up properly has given me a lot less headaches.

    source
  • Eldritch@piefed.world ⁨6⁩ ⁨months⁩ ago

    And I generally enjoy Veronica's presentation. Knowledgable and simple.

    source

    • mesamunefire@piefed.social ⁨6⁩ ⁨months⁩ ago

      Her https://tinkerbetter.tube/w/ffhBwuXDg7ZuPPFcqR93Bd made me learn a new way of looking at data. There was some tricks I havent done before. She has such good videos.

      source

      • -> View More Comments