Introducing Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration

Comments

• Shepy@feddit.uk ⁨1⁩ ⁨year⁩ ago

  This sounds amazing, definitely going to add this to my servarr setup next few days.

  source
• Dasnap@lemmy.world ⁨1⁩ ⁨year⁩ ago

  It’s only once you install something like this that you realize just how many torrents are porno.

  source

  • Salamendacious@lemmy.world ⁨1⁩ ⁨year⁩ ago

    scrubs YouTube clip

    source

    • PipedLinkBot@feddit.rocks [bot] ⁨1⁩ ⁨year⁩ ago

      Here is an alternative Piped link(s):

      scrubs YouTube clip

      Piped is a privacy-respecting open-source alternative frontend to YouTube.

      I’m open-source; check me out at GitHub.

      source
• droopy4096@lemmy.ca ⁨1⁩ ⁨year⁩ ago

  @mgdigital, first thing I’be noticed: reliance on “heavier” database stack (pg + redis), at least from the first glance at docker-compose. My suggestion would be to have an option for minimalist setup with sqlite and without redis if possible. That would work better for those of us flying with minimal hardware (rpi, old PC and such).

  source

  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Hi, this is a great point and one that I’ve already given consideration to. I’ll address separately the issue of the primary datastore ,i.e. Postgres, and the Redis dependency:

    Postgres as the only option for the data store

    There are 2 reasons for this:

    • Performance: while SQLite could offer a simpler/embedded data store, it simply doesn’t have the performance and features of Postgres. Bitmagnet has a faceted search engine and is write-intensive (it will be discovering ~5k torrents per hour and writing these to the database along with associated metadata). As such, its database may not be suitable for running on older hardware. A SQLite adapter, if it was developed, may simply not be up to the job (although as I haven’t attempted this I can’t say what the performance would be like). That said, Bitmagnet itself is not especially resource intensive, you could probably run it on a Raspberry PI but point it to a Postgres instance on some more powerful hardware. At this stage I’ve only been running it on a M2 Mac Mini with Postgres located on its SSD and so would be interested to know people’s mileage on other hardware.
    • Development, support and maintenance overhead: I’m a lone developer and this project is already too big for one person. A SQLite adapter, if feasible performance-wise, I think could only happen if other contributors joined the project as my to-do list is already pretty long. It would have to achieve feature parity with the Postgres implementation which makes use of several Postgres-specific features and extensions. It would also mean a longer testing cycle and therefore probably a slower release cadence. That said, if there was enough demand and assistance then I’d be open to looking into the feasibility of this once the rest of the application is a little more mature and the current database schema more finalised.

    Redis dependency

    Redis is currently used only for the asynchronous task queue. I would like to have put this in Postgres, but there simply is not a good out-of-the-box solution that works well with Postgres and GoLang, and is actively maintained. I looked at quite a few queuing libraries and eventually settled on asynq (github.com/hibiken/asynq), which is a great library and does the job well - but could really do with support for non-Redis backends.

    Using Redis here was a pragmatic decision that allowed me to make progress, rather than an optimal one. I guess I could have built a simple Postgres-based queue myself but that would have been a distraction and probably sub-optimal compared with a mature/separately developed library. It remains an option. Since I looked into this a new project has sprung up which I’m keeping an eye on - www.tork.run - it has a Postgres backend and looks like it might be up to the job, but is very new.

    So yes, I’m very aware that the additional Redis dependency is not ideal and it may well disappear at some point.

    source

    • mlunar@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Hi, those points are certainly valid and I have nothing against these picks!

      I just wanted to chime in that perf might not be as big of a problem as you might expect. 5k/hour is 1.4/sec, which sqlite should for sure be able to handle.

      In fact, you can do hundreds to thousands of writes/sec, as long as you batch them in transactions (as by default each query is executed in its own transaction).

      source
    • droopy4096@lemmy.ca ⁨1⁩ ⁨year⁩ ago

      thank you for such a detailed response. I would love to contribute however at the moment my capacities are rather limited but otherwise I’d be willing to add sqlite adapter. From your description it sounds like currently architecture is narrowly locked on PostgreSQL features. In my daily job I love PostgreSQL for big apps and stacks but I’m also aware how “hungry” PG can be, which is why I’m wondering whether it’s “too big of a hammer” for this particular problem. Also, setting up single service is easier to novices vs maintaining several. Docker compose is nice but it has it’s limitations.

      source
• thantik@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Very nice. This gets rid of any questionably legal gray area of using sites like Nyaa, etc for Torrent links. Also provides a bit of robustness against censorship when those sites get taken down. Looks like I’m gonna have to set up proxmox on a machine this weekend, as Windows sucks dick for docker containers and that’s what I’ve got most of my *arr stuff hosted on currently.

  It’ll be a good thing anyways, as most of those instances aren’t running through my VPN yet and I should just centralize them on proxmox and run all the torrents, etc through containerized instances for security.

  source

  • kungen@feddit.nu ⁨1⁩ ⁨year⁩ ago

    This gets rid of any questionably legal gray area of using sites like Nyaa, etc for Torrent links

    Except that now you’re asking the swarm for metadata behind a boatload of info_hashes? Unlikely anyone would care (though you’d be surprised how many DMCAs I get when just having a simple open tracker running, not even an indexet), but I don’t see it as being any less grey than using any existing sites.

    source

    • thantik@lemmy.world ⁨1⁩ ⁨year⁩ ago

      In some jurisdictions hosting links to pirated content is considered illegal. In others it is not.

      source

      • -> View More Comments
• ryannathans@aussie.zone ⁨1⁩ ⁨year⁩ ago

  Does it infiniely crawl, storing all metadata about every torrent it finds forever?

  source

  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Yep

    source
• 726a67@lemmy.sdf.org ⁨1⁩ ⁨year⁩ ago

  Looks super interesting; starred!

  source
• spiritedpause@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

  Dude this is amazing! Exactly the sort of thing I’ve been hoping would pop up to further “decentralize” the torrent search experience.

  So I’m trying to run it on my machine through the docker-compose option, and I’m seeing something weird. It shows as successfully running, but when I go to the port it should be running on, I get “unable to connect” on my browser.

  When I check my containers running, it shows the 3 bitmagnet containers, but the port doesn’t show.

  i.imgur.com/D4R1Le5.png

  source

  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Hi, the default port is 3333, which should be exposed if you’re using the example configuration here: bitmagnet.io/setup/installation.html - I’m not sure what the app is in your screenshot but the provided config definitely exposes that port and is tested on Docker for Mac.

    source

    • spiritedpause@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

      Just pulled the latest and tried again, and it works now! Thanks

      source
• prim3r@lemmy.ca ⁨1⁩ ⁨year⁩ ago

  This looks really cool! How resource intensive is this? What sort of storage requirements are there for this to be a reasonably reliable method of acquiring media? I’m probably just gonna find out myself. I’ve recently fully switched over to usenet, but this could make torrents pretty compelling again.

  source

  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Hi, and thanks!

    As a priority I’d like to gather some more rigorous performance benchmarks, but I can give you some hand-wavey stats now: Bitmagnet is currently fluctuating between 2-10% CPU usage on my M2 Mac Mini, and is using ~120MB of memory having currently been running for around 48 hours. Overall, the GoLang implementation seems pretty efficient to me considering how much I know is going on in the background.

    Disk space usage of the database- this will be highly dependent on 2 configuration options, the first of which I’ve only just added in the just-released version. Copied from the configuration page of the website:

    • dht_crawler.save_files (default: true): If true, file metadata from the DHT crawler will be saved to the database. This provides more rich information about a torrent, but will use a lot more disk space. If disk space is at a premium you may want to consider disabling this.
    • dht_crawler.save_pieces (default: false): If true, the DHT crawler will save the pieces bytes from the torrent metadata. The pieces take up quite a lot of space, and aren’t currently very useful, but they may be used by future features.

    For me, 24 hours of crawling uses ~2.5GB of database disk space for metadata on the ~120k torrents it has discovered. Yep, that sounds like a lot, however 90% of that is taken up with the files metadata, and could have been saved by setting dht_crawler.save_files to false. In fact I may set this to false by default and allow users to opt-in to the full-fat torrent info.

    I’ve also imported the entire RARBG backup (the SQLite one, see tutorial on the Bitmagnet website). This, along with all the associated metadata from TMDB, took around 4GB of database space, which seems quite acceptable considering it’s basically every movie and TV show. Note that this does NOT include the metadata on individual files as I described above.

    A priority feature for me (detailed on website) is smart deletion - this would allow you to automatically discard a lot of data that can be automatically determined of no interest and therefore greatly reduce disk space demands.

    source
  • kautau@lemmy.world ⁨1⁩ ⁨year⁩ ago

    As some interested in Usenet, what’s the best provider and client to start with in your opinion?

    source

    • prim3r@lemmy.ca ⁨1⁩ ⁨year⁩ ago

      I’ve been using easynews/nzbgeek/nzbget with an arr stack on debian and it’s worked well for me. I’m fairly new to usenet, so take this with a giant grain of salt.

      source

      • -> View More Comments
  • deafboy@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Running for 6 days, save_pieces: false

    My database is currently 184 GB

    source
• pete_the_cat@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Sounds interesting 😀 I’ll keep an eye on it, though I won’t be a primary user, I switched to usenet about a decade ago and only use torrents as a last resort.

  source
• Shdwdrgn@mander.xyz ⁨1⁩ ⁨year⁩ ago

  Looks like a fun project, but will you be providing any info on setting it up from scratch? I just don’t have an interest in docker containers.

  source

  • cyberpunk007@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Out of curiosity, why not? I’ve come around.

    source

    • Shdwdrgn@mander.xyz ⁨1⁩ ⁨year⁩ ago

      I’ve just always used VMs for everything and set up each service to match my existing system. For example, my postfix servers have to all tie in to LDAP, mailman, and the host of services for authenticating email. It seems like the point of docker is to just have a completely preconfigured and self-contained setup. I guess I Just don’t see how that would work in my environment where I already have some services like databases or LDAP already running elsewhere, and I run multiple instances for redundancy. And if I have to reconfigure all that stuff in docker anyway, how is that any better than simply using my existing VMs?

      source

      • -> View More Comments
    • Dasnap@lemmy.world ⁨1⁩ ⁨year⁩ ago

      I personally love containers (probably because I use them for work) but I can understand someone not wanting another layer of abstraction if they’ve worked bare-metal for a long time.

      source
  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Hi, yes this is mentioned on the installation page of the website, below the Docker instructions. The app can be installed Dockerless using go install; if you choose this option you’ll have to provide and configure Postgres and Redis instances for the app to connect to. That said, Docker is the recommended and easiest option.

    source

    • Shdwdrgn@mander.xyz ⁨1⁩ ⁨year⁩ ago

      I saw that, but didn’t recognize the ‘go’ command as anything available on Debian. Just did some quick digging though and now I see it’s a new language and I believe I have an idea how to get it installed for compiling so I will give that a shot.

      source

      • -> View More Comments
• Decronym@lemmy.decronym.xyz ⁨1⁩ ⁨year⁩ ago

  Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

  Fewer Letters	More Letters
  Plex	Brand of media server package
  SSD	Solid State Drive mass storage
  VPN	Virtual Private Network

  ---

  [Thread #191 for this sub, first seen 5th Oct 2023, 14:25] [FAQ] [Full list] [Contact] [Source code]

  source
• gentoo_biscuit@thelemmy.club ⁨1⁩ ⁨year⁩ ago

  This sounds awesome, I’ll give it a try! Would this work in i2p?

  source

  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I’ve never used I2P but I don’t see why not!

    source
• BlueEther@no.lastname.nz ⁨1⁩ ⁨year⁩ ago

  seems to work well

  just one question, is it expected to have 10,000 out of 12,000 as unknown?

  source

  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Hi, yep that’s expected. Torrents will only move out of “Unknown” once the classifier is able to categorise them. The classifier currently only supports movie and TV show content, and can recognise these with quite high accuracy assuming a well-named torrent (and a badly named torrent is unlikely to be a high quality release). The other content types (movies, games etc) can currently only be populated via an import (see the tutorial on the website). A priority feature is classifiers for other content types - however we will likely always have a lot of torrents ending up in “Unknown” given the poor naming of many crawled items. Another roadmap feature, smart deletion, could help in future with getting rid of all the rubbish whose contents cannot be inferred from the torrent name.

    source
• railsdev@programming.dev ⁨1⁩ ⁨year⁩ ago

  You had me GraphQL 🥰

  source
• Molecular0079@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Is it safe to run this without a VPN if I am just using it to index?

  source
• snekerpimp@lemmy.world ⁨1⁩ ⁨year⁩ ago

  That was easy to set up. I’ll let it churn for a few days and we’ll see how helpful it is. Thanks for the app.

  source
• Skeptic043@kbin.social ⁨1⁩ ⁨year⁩ ago

  Being relatively new to the self hosted experience and still working through how everything in an arr setup interacts, along with what issues can occur and how to troubleshoot/fix them, this sounds incredibly useful. I'll definitely be looking into integrating this into my own setup and providing feedback when I can!

  source
• mctoasterson@reddthat.com ⁨1⁩ ⁨year⁩ ago

  Maybe I’m misunderstanding but wouldn’t it just be easier to use a good private tracker, assuming you can get an invite?

  source

  • lud@lemm.ee ⁨1⁩ ⁨year⁩ ago

    Yes, of course.

    source
• palitu@aussie.zone ⁨1⁩ ⁨year⁩ ago

  Very cool!

  source
• themoonisacheese@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

  I use magnetico and have no need for the bells and whistles, but that seems really interesting!

  source
• Willdrick@lemmy.world ⁨1⁩ ⁨year⁩ ago

  This looks kinda neat, I even tore down my whole servarr stack to give it a go, alas I can’t get bitmagnet to “talk” with prowlarr. I’m probably doing something really stupid, but I can’t figure out how to add the whole thing under a single docker network, I get errors like network somename was found but has incorrect label com.docker.compose.network set to “”

  source
• LienNoir@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Hi, am i missing something, the bitmagnet image keep restarting when i check with “docker ps”, the other 2 containers are working as intended. And port 3333 doesn’t show anything.

  source

  • mgdigital@lemmy.world ⁨1⁩ ⁨year⁩ ago

    There’s a PR currently open for multi-platform builds so should have this sorted soon

    source
  • emhl@feddit.de ⁨1⁩ ⁨year⁩ ago

    What are your logs showing? docker logs -f bitmagnet

    source

    • LienNoir@lemmy.world ⁨1⁩ ⁨year⁩ ago

      log: exec /bitmagnet: exec format error

      I am on ARM (pi4) maybe it’s the issue

      source

      • -> View More Comments