Do we need a crawler to load content from other lemmy instances? How are we supposed to federate without one?

Submitted ⁨⁨5⁩ ⁨days⁩ ago⁩ by ⁨uhdeuidheuidhed@thelemmy.club⁩ to ⁨fediverse@lemmy.world⁩

I’m trying to start my own lemmy instance, and reading through the documentation at join-lemmy.org/…/federation_getting_started.html it looks like we need to manually search for just about everything that can be found on other lemmy instances.

It works. I’m able to search for communities/comments/posts and then my instance will load only that community/comment chain/post and only after I’ve searched it. No updates.

Is this what we’re supposed to be doing? Seems pretty tedious to have someone that’s supposed to go to other instances and then search for all of their content on my instance in order to federate with it.

I have a feeling there is an easier way, but I’m legitimately not seeing it.

Right now, it looks like the only way users on my instance will get to see content from other instances is if I manually search for just about everything they’ll get to see.

source

Comments

Sort:hotnew top

FrostyTrichs@crazypeople.online ⁨5⁩ ⁨days⁩ ago
lemmy-federate.com

source
- uhdeuidheuidhed@thelemmy.club ⁨5⁩ ⁨days⁩ ago
  Thanks, this looks interesting.
  
  I tried adding my instance under “Manage Instances” but got this error: Request timed out: GET https://{IP}/nodeinfo/2.1
  
  I’m not sure what this means. Could anyone help provide an explanation?
  
  source
  - FrostyTrichs@crazypeople.online ⁨5⁩ ⁨days⁩ ago
    Sorry but I’m not sure. Every time I’ve used that tool the setup has gone flawlessly. The first time you use it you put in your instance URL and it should output something for you to add to your DNS records. If you just made the DNS changes it can take a few minutes to several hours for those changes to be pushed out.
    
    source
    -> View More Comments
- Coelacanth@feddit.nu ⁨5⁩ ⁨days⁩ ago
  This is the answer, I hope it’s common knowledge by now.
  
  source
jollyroberts@jolly-piefed.jomandoa.net ⁨5⁩ ⁨days⁩ ago
Piefed has a couple of different ways to populate a new server. Pulling lemmyverse data and subscribing to comms that way, or scanning a remote Lemmy/mbin/pieces server for comms. You can look at that code if you want inspiration.

source
tofu@lemmy.nocturnal.garden ⁨5⁩ ⁨days⁩ ago
That’s basically how it’s done. Searching for them isn’t enough I think, in order to get updates, a user from your instance needs to subscribe, so a crawler might create a list of communities, but it won’t see news.

Yes, starting a new instance is tedious work.

source
Steve@communick.news ⁨5⁩ ⁨days⁩ ago

Right now, it looks like the only way users on my instance will get to see content from other instances is if I manually search for just about everything they’ll get to see.

They can do that themselves.

source
vk6flab@lemmy.radio ⁨5⁩ ⁨days⁩ ago
Make sure you’re aware of any legal liability implications for content posted by your users.

source
frongt@lemmy.zip ⁨5⁩ ⁨days⁩ ago
There’s a bot you can use to jump-start federation. No idea what it’s called, but I’m sure you can find it with a few seconds of searching.

source
squaresinger@lemmy.world ⁨5⁩ ⁨days⁩ ago
That’s how federation is supposed to work. If one user searches content, you get it once. If a user of your instance subscribes to a community, you get updates for that community.

The point is to reduce load on your instance by not federating content that nobody on your instance needs.

It’s based on a flawed understanding of how communities work, specifically that reddit-style forums only work well when you have huge communities that everyone sibscribes to. So, most instances end up with most content anyway.

Also, keep in mind, all conent that has been federated to your instance is on your srver and thus legally your responsibility. If there’s illegal content on there, you are liable if you don’t delete it.

source