Comment on introducing copyparty, the FOSS file server

chellomere@lemmy.world ⁨8⁩ ⁨months⁩ ago

• suzune@ani.social ⁨8⁩ ⁨months⁩ ago

  Just a remark from someone who runs ZFS since the beginning. Many people don’t like the deduplication feature because of its memory footprint.

  It’s also nice to have this feature without relying on a certain filesystem.

  source

  • chellomere@lemmy.world ⁨8⁩ ⁨months⁩ ago

    I assume you mean automatic deduplication? I haven’t used ZFS, but BTRFS does not have that. There are a variety of ways to perform deduplication, I have duperemove scheduled to run regularly.

    If ZFS is still capable of being instructed to perform deduplication when automatic deduplication is turned off, which it really should be able to do, then this should work even with it turned off.

    source
• tripflag@lemmy.world ⁨8⁩ ⁨months⁩ ago

  BTRFS and ZFS support real deduplication via copy on write, and would eliminate all current disadvantages of symlink and hardlink deduplication. It just works.

  yeah that’s a good point, I’ll add an option to take advantage of this if you know you’re running on a filesystem where that works as intended.

  Why have it be one huge python source file?

  oh don’t worry, it’s all separate files during development – there’s a build-stage which bundles everything up into a single file for distribution. But thanks for the concern :D

  source

  • RecallMadness@lemmy.nz ⁨8⁩ ⁨months⁩ ago

    What do you use to bundle into one file?

    source

    • tripflag@lemmy.world ⁨8⁩ ⁨months⁩ ago

      copyparty-sfx.py is a custom packer (see this reply) created by make-sfx.sh, and copyparty.pyz is a standard zipapp, created by make-pyz.sh. The zipapp has more disadvantages than the sfx.py, so that’s the default/recommended build.

      source
  • chellomere@lemmy.world ⁨8⁩ ⁨months⁩ ago

    Ah, so you have compiled it into one file? Didn’t know that was possible for python, what tool do you use for this?

    source

    • tripflag@lemmy.world ⁨8⁩ ⁨months⁩ ago

      sooo this is one of the things that started with someone saying “wouldn’t it be funny if…”

      if you open copyparty-sfx.py in a text editor, you’ll see how – but please make sure to use an editor which is able to handle about 600 KiB of comments which contain invalid utf8 / binary garbage 😁

      I ended up rolling my own packer since I wanted optimal encoding efficiency, and everything I could find would do stuff like base85 or ucs2 tricks, but it turns out python is perfectly happy with binary garbage in comments if you declare that the file is latin-1 so it realizes all hope is lost :D

      the only drawback of the sfx.py is that it needs to extract to $TEMP before running, so that’s the slight advantage of the zipapp (the .pyz alternative), but that suffers from some performance reduction in return, and is more hermetic (doesn’t let you swap out the bundled dependencies with fresh versions as easily if necessary)

      source

      • tatterdemalion@programming.dev ⁨8⁩ ⁨months⁩ ago

        Why not use zipapp?

        docs.python.org/3.13/library/zipapp.html

        source
      • chellomere@lemmy.world ⁨8⁩ ⁨months⁩ ago

        Ah, reminds me of the old self-extracting gzip executable trick. I used that once a very long time ago to make a 4k linux intro, before I realized to be competitive I should switch to windows to be able to use Crinkler, which is superior even though the decompressor is part of the executable.

        source